Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

2,457 views

Published on

End-user’s requirements for secure IT products are continually increased in environment that are affected directly to human life and industry such as IoT, CPS. Because vendors and end-user sell or buy products based on trustworthy or objective security evaluation results, security evaluation roles are important. Security Evaluations are divided to two parts, one is evaluation on design level such as ISO/IEC 29128(Verification of Cryptographic Protocols) and another one is post-implementation level such as ISO/IEC 15408(Common Criteria). These security evaluation standards, both ISO/IEC 29128 and ISO/IEC 15408, advise to use formal verification and automated tools when high assurance level of target products is required.
For a long time, vulnerability detection using automated tools have been tried and studied by many security researchers and hackers. And recently, the study related to automated vulnerability detection are now more active than ever in hacking community with DARPA’s CGC(Cyber Grand Challenge). But, too many tools are developed continually and usually each tool has their own purpose to use, so it’s hard to achieve ultimate goal of security evaluation effectively and verify evaluation results.
Furthermore, there are no references for categorizing about automated tools on perspective of security evaluations. So, in this presentation we will list up, categorize and analyze all of automated tools for vulnerability detection and introduce our result such as pros and cons, purpose, effectiveness, etc.

-- InHyuk Seo
My name is Inhyuk Seo(Nick: inhack). I graduated B.S. in Computer Science and Engineering at Hanyang University(ERICA) in 2015. Now I’m a researcher and M.S. of SANE(Security Analaysis aNd Evaluation) Lab at Korea University. I’m interested in Programming Language, Software Testing, Machine Learning, Artificial Intelligence.
In 2012, I completed high-quality information security education course “the Best of the Best(BoB)” hosted by KITRI(Korea Information Technology Research Institute) and conducted “Exploit Decoder for Obfuscated Javascript” Project.
I participated in many projects related with vulnerability analysis. I conducted “Smart TV Vulnerability Analysis and Security Evaluation” and “Developing Mobile Security Solution(EAL4) for Military Environment ”. Also, I participated in vulnerability analysis project for IoT products of various domestic tele-communications.

-- Jisoo Park
Jisoo Park graduated with Dongguk University B.S in Computer science engineering. He participated in secure coding research project in Programming Language Lab and KISA(Korea Internet & Security Agency). He worked as a software QA tester at anti-virus company Ahnlab. He also completed high-quality information security education course “Best of the Best” hosted by KITRI(Korea Information Technology Research Institute) and conducted security consulting for Car sharing service company.
Now, Jisoo Park is a

Published in: Technology
  • Be the first to comment

  • Be the first to like this

[CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

  1. 1. Inhyuk Seo(inhack), Jisoo Park(J.Sus), Seungjoo Kim SANE(Security Analysis aNd Evaluation) Lab Korea University(高麗大學校) Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness
  2. 2. Contents • Who are we? • Introduction • Security Engineering, the Way to Information Assurance • High-Assurance, the Key of CPS • Tools for Security Testing & Evaluation - Tools for Design Assurance / Tools for Code Assurance • Demo (Design / Code) • Conclusion • Acknowledgement • Q&A • Reference
  3. 3. Who are we? Inhyuk Seo (徐寅赫) E-mail : jisoo8881@korea.ac.kr Jisoo Park received his B.S (2015) in Computer Science Engineering from Dongguk University in Korea. He worked at antivirus company Ahnlab as S/W QA trainee for 6 month. Also he completed high-quality information security education course “Best of the Best” hosted by KITRI(Korea Information Technology Research Institute). Now, He is a M.S course student at CIST SANE Lab, Korea University and interested in Common Criteria, Security Engineering(Especially Threat modeling). Jisoo Park (朴志洙) E-mail : inhack@korea.ac.kr My name is Inhyuk Seo(Nick: inhack). I graduated B.S. in Computer Science and Engineering at Hanyang University(ERICA) in 2015. Now I’m a researcher and M.S. of SANE(Security Analaysis aNd Evaluation) Lab at Korea University. In 2012, I completed high-quality information security education course “the Best of the Best(BoB)” hosted by KITRI(Korea Information Technology Research Institute) and participated in many projects related with vulnerability analysis. I’m interested in Programming Language, Software Testing, Machine Learning, Artificial Intelligence.
  4. 4. Seungjoo Gabriel Kim (金昇柱) E-mail: skim71@korea.ac.kr Homepage : www.kimlab.net Facebook, Twitter : @skim71 Prof. Seungjoo Gabriel Kim received his B.S, M.S and Ph.D. from Sungkyunkwan University(SKKU) of Korea, in 1994, 1996, and 1999, respectively. Prior to joining the faculty at Korea University (KU) in 2011, he served as Assistant & Associate Professor at SKKU for 7 years. Before that, he served as Director of the Cryptographic Technology Team and the (CC-based) IT Security Evaluation Team of the Korea Internet & Security Agency(KISA) for 5 years. He is currently a Professor in the Graduate School of Information Security Technologies(CIST). Also, He is a Founder and Advisory director of hacker group, HARU and an international security & hacking conference, SECUINSIDE. Prof. Seungjoo Gabriel Kim’s research interests are mainly on cryptography, Cyber Physical Security, IoT Security, and HCI Security. He is a corresponding author. Who are we?
  5. 5. Intro Level of trust that it really does! Assurance The User’s degree of trust in that information Information Assurance
  6. 6. Intro Rise of the Information Assurance Gulf War has often been called the first information war. “The harbinger of IA” 1991 U.S. DoD Directive 5-3600.1 : The first standardized definition of IA 1996 Information Security (INFOSEC) Era 1980 ~ “The communication network that supported Operation Desert Storm was the largest joint theater system ever established. It was built in record time and maintained a phenomenal 98 percent availability rate. At the height of the operation, the system supported 700,000 telephone calls and 152,000 messages per day. More than 30,000 radio frequencies were managed to provide the necessary connectivity and to ensure minimum interference.” Debra S. Herrmann, “Security Engineering and Information Assurance”
  7. 7. Intro Information Assurance “Measures that protect and defend information and information systems by ensuring their availability, integrity, authentication, confidentiality, and non- repudiation. This includes providing for restoration of information systems by incorporating protection, detection, and reaction capabilities.” DoD Directive 8500.01E Rise of the Information Assurance Gulf War has often been called the first information war. “The harbinger of IA” 1991 U.S. DoD Directive 5-3600.1 : The first standardized definition of IA 1996 Information Security (INFOSEC) Era 1980 ~
  8. 8. What are the differences between Information Security and Information Assurance? Intro
  9. 9. Intro Information Security (情報保護) Information Assurance (情報保證) Dates Since 1980s Since 1998 Subject of protection Information and Information system Business as a whole Goal Confidentiality, Integrity, Availability Confidentiality, Integrity, Availability, Non- repudiation, Accountability, Auditability, Transparency, Cost-effectiveness, Efficiency Type of information Primarily electronic All types Approach Domination of the technical approach, initial attempts to consider soft aspects All-encompassing multi-disciplinary systematic approach Security Mechanism Primary focus is on technical security mechanism; initial consideration of organizational and human-oriented mechanism All available (technical, organizational, human-oriented, legal) Role within a business Supporting system, often inducing some restrictions on business An integral aspect of business, business enabler Flow of security decision Bottom-Top Top-Bottom
  10. 10. Intro Information Security (情報保護) Information Assurance (情報保證) Dates Since 1980s Since 1998 Subject of protection Information and Information system Business as a whole Goal Confidentiality, Integrity, Availability Confidentiality, Integrity, Availability, Non- repudiation, Accountability, Auditability, Transparency, Cost-effectiveness, Efficiency Type of information Primarily electronic All types Approach Domination of the technical approach, initial attempts to consider soft aspects All-encompassing multi-disciplinary systematic approach Security Mechanism Primary focus is on technical security mechanism; initial consideration of organizational and human-oriented mechanism All available (technical, organizational, human-oriented, legal) Role within a business Supporting system, often inducing some restrictions on business An integral aspect of business, business enabler Flow of security decision Bottom-Top Top-Bottom Protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction Validating that the information is authentic, trustworthy, and accessible
  11. 11. Security Engineering, the Way to Information Assurance
  12. 12. What is Information Assurance’s Goal? Security Engineering
  13. 13. Security Engineering Goal of Information Assurance Dependability The ability of the system to deliver services when requested Availability The ability of the system to deliver services as specified Reliability The ability of the system to protect itself against accidental or deliberate intrusion The ability of the system to operate without catastrophic failure Safety Security Reflect the extent of the user’s confidence that it will operate as users expects that it will not ‘fail’ in normal use
  14. 14. Domain Reliability Security Safety Financial System Medium High No DB of Medical Records Medium Medium Medium Air Traffic Control System Medium High High Automobile High Medium High Defcon 23 – Charlie Miller & Chris Valasek “Remote Exploitation of an Unaltered Passenger Vehicle” It was ‘Low’ at first, Security Engineering Goal of Information Assurance
  15. 15. How can we achieve Information Assurance? Security Engineering
  16. 16. How can we achieve Information Assurance? Security Engineering Security Engineering
  17. 17. Security Engineering is about building systems to remain dependable in the face of malice, error and mischance. As a discipline, it focuses on the tools, needed to design, implement and test complete systems and to adapt existing systems as their environment evolves. – Ross Anderson, Computer Laboratory in University of Cambridge - What is Security Engineering? Security Engineering
  18. 18. Policy Assurance Mechanisms Policy Assurance Design Assurance Implementation Assurance Operational Assurance Assurance needed at all stage of System life cycle Ultimate Goal of Security Engineering Security Engineering What is Security Engineering?
  19. 19. Requirements Design Implementation Release Maintenance System Engineering Life Cycle Process (ISO/IEC/IEEE 15288 : 2015) • Business or Mission Analysis • Stakeholder Needs and Requirements Definitions • System Requirements Definition • Architecture Definition • Design Definition • System Analysis • Implementation • Integration • Verification • Transition • Validation • Operation • Maintenance • Disposal Security Engineering What is Security Engineering? Provide Security Engineering throughout the Life Cycle
  20. 20. Case Study : Microsoft Security Development Life Cycle Security Engineering
  21. 21. Case Study : Microsoft Security Development Life Cycle Does it really work? 34 3 187 SQL Server 2000 SQL Server 2005 Competing commercial DB Total Vulnerabilities Disclosed 36 Month after Release 46% reductio n 119 66 400 242 157 Windows XP Windows Vista OS A OS B OS C Total Vulnerabilities Disclosed On year after Release 46% reduction After SDLBefore SDL After SDLBefore SDL 91% reduction Analysis by Jeff Jones(Microsoft technet security blogWindows Vista One year Vulnerability Report, Microsoft Security Blog 23 Jan 2008 Security Engineering
  22. 22. High-Assurance, the Key of CPS
  23. 23. High Assurance, the Key of CPS What is “High-Assurance”? High-Assurance means that it can be mathematically proven that the system works precisely as intended and designed. and High-Assurance development means that there are clear and compelling evidences in each development phase.
  24. 24. What is “CPS”? Cyber Physical Systems(CPS) are co- engineered interacting network of physical and computational components. CPS will provide the foundation of our critical infrastructure, form the basis of emerging and future smart services, and improve our quality of life in many areas. Internet of Things Cyber Physical System AssuranceSecurity VS High Assurance, the Key of CPS
  25. 25. Where “High-Assurance” needed Information Assurance Security Engineering Critical Infrastructure Finance Aviation Government Medical Automotive Railway Energy . . High- Assurance Apply & Guarantee High Assurance, the Key of CPS
  26. 26. Some standards or regulations for critical infrastructure are not enough for achieving dependability. • Most of them don’t have Security feature. Domain Standard / Regulation Road Vehicles ISO 26262 Aviation DO-178B, 178C, 254, 278A …. Medical IEC 62304 Railways EN 50128 High Assurance, the Key of CPS
  27. 27. ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security” ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability” Standard / Regulation Assurance Level ISO 26262 ASIL A ASIL B ASIL C ASIL D DO-254 DAL E DAL D DAL C DAL B DAL A ISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4 ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7 HighLow High Assurance, the Key of CPS
  28. 28. ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security” ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability” Standard / Regulation Assurance Level ISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4 ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7 HighLow High Assurance, the Key of CPS
  29. 29. Example : ISO/IEC 29128 Verification of Cryptographic Protocol Protocol Assurance Level PAL1 PAL2 PAL3 PAL4 Protocol Specification Semiformal description of protocol specification Formal description of protocol specification Formal description of protocol specification in a tool-specific specification language, whose semantics is mathematically defined Adversarial Model Security Property Self-assessment evidence Informal argument or mathematically formal paper-and-pencil proof that the cryptographic protocol satisfies the given objectives and properties with respect to the adversarial model Tool-aided bounded verification that the specification of the cryptographic protocol satisfies the given objectives and properties with respect to the adversarial model Tool-aided unbounded verification that the specification of the cryptographic protocol satisfies the given objectives and properties with respect to the adversarial model Tool-aided unbounded verification that the specification of the cryptographic protocol in its adversarial model achieves and satisfies its objectives and properties. High Assurance, the Key of CPS
  30. 30. Example : Common Criteria ISO/IEC 15408 Evaluation criteria for IT security Evaluation Assurance Level Description EAL 7 Formally verified design and tested EAL 6 Semiformally verified design and tested EAL 5 Semiformally designed and tested EAL 4 Methodically designed, tested, and reviewed EAL 3 Methodically tested and checked EAL 2 Structurally tested EAL 1 Functionally tested Gerwin Klein, Operating System Verification – An Overview High Assurance, the Key of CPS
  31. 31. Example : Common Criteria ISO/IEC 15408 Corresponding assurance levels in ISO/IEC 29128 High Assurance, the Key of CPS
  32. 32. How to Get it? • Measurable & Mathematically provable  Formal Verification • By using Tools High Assurance, the Key of CPS
  33. 33. How to Get it? Established in March 2012, as a Research Association, which headquarters is located in Tagajo City of Miyagi Prefecture. CSSC’s testbed is composed of 9-types of simulated plants and it is capable to organize cybersecurity hands-on exercises which simulate cyber attack Control System Security Center (CSSC) Major operation plans – System security verification High Assurance, the Key of CPS
  34. 34. How to Get it? “The goal of the HACMS program is to create technology for the construction of high-assurance cyber- physical systems, where high assurance is defined to mean functionally correct and satisfying appropriate safety and security properties.” Dr. Raymond Richards, Information Innovation Office Program Manager of HACMS High-Assurance Cyber Military System (HACMS) High Assurance, the Key of CPS
  35. 35. Tools for Security Testing & Evaluation
  36. 36. Tools for Security Testing & Evaluation Automation Tools for Hacker & Bug Hunters • Automation Vulnerability Detection Tools developed by hacker/bug hunter are only for the purpose of finding 0-day (Unknown Vulnerability) easily. Automation Tools for Evaluation Ultimate goal of Security testing & evaluation There are no mistakes in security testing process and Guarantee objective analysis reports or evaluation results Independent from evaluator’s capability or expertise. So anyone who uses the same tools should be able to make same results.
  37. 37. What should we consider when we choose Automated security testing tools in evaluation? Tools for Security Testing & Evaluation
  38. 38. Assessment Features for Automated Tools User-Friendly Effectiveness Scalability Tools for Security Testing & Evaluation
  39. 39. Tools for Design Assurance
  40. 40. Tools for Design Assurance Assessment items to choose Automated Tools for Design Assurance (1) User-Friendly • Usability • Analysis Report • Requirement to Evaluator (Expertise, Background Knowledge) (2) Effectiveness • Automation Level • Model Description Method • Licensing & Cost (3) Scalability • Supported Platforms
  41. 41. Cryptographic Protocol Model Checking Theorem Proving Based • NRL • FDR • SCYTHER • ProVerif • AVISPA(TA4SP) • CryptoVerif • EBMC ……. • Isabelle/HOL • BPW • Game-based Security Proof • VAMPIRE • ……. Tools for Design Assurance
  42. 42. Tools for Design Assurance Cryptographic Protocol (Model Checking) • The Maude NRL Protocol Analyzer (Maude-NPA) Assessment Items Description Usability GUI(Graphic User Interface) Analysis Report O Requirement to Evaluator Protocol Design & Modeling Ability Automation Level Interactive Model Description Method Maude-PSL (Maude Protocol Specification Language) Licensing & Cost Non-Commercial (University of Illinois) Supported Platform Mac OS X
  43. 43. Cryptographic Protocol (Model Checking) • FDR(Failure-Divergence-Refinement) Assessment Items Description Usability GUI Analysis Report O Requirement to Evaluator Protocol Design & Modeling Ability Automation Level Interactive Model Description Method Formal Language (CSP) Licensing & Cost Non-Commercial (University of Oxford) Supported Platform Linux / Mac OS X Tools for Design Assurance
  44. 44. Cryptographic Protocol (Model Checking) • Syther Assessment Items Description Usability GUI Analysis Report O Requirement to Evaluator Protocol Design & Modeling Ability Automation Level Interactive Model Description Method SPDL (Standard Page Description Language) Licensing & Cost Non-Commercial (University of Oxford) Supported Platform Linux / Windows / Mac OS X Tools for Design Assurance
  45. 45. Cryptographic Protocol (Model Checking) • ProVerif Assessment Items Description Usability CLI (but Easy to Use) Analysis Report O Requirement to Evaluator Protocol Design & Modeling Ability Automation Level Interactive Model Description Method PV Script (ProVerif Script) Licensing & Cost Non-Commercial (PROSECCO) Supported Platform Linux / Windows / Mac OS X Tools for Design Assurance
  46. 46. Cryptographic Protocol (Theorem Proving) • Isabelle/HOL(Higher-Order Logic) Assessment Items Description Usability GUI, IDE(Integrated Development Environment) Analysis Report O Requirement to Evaluator Protocol Design & Modeling Ability Automation Level Interactive Model Description Method Functional & Logic Language (HOL) Licensing & Cost Non-Commercial (University of Cambridge) Supported Platform Linux / Windows / Mac OS X Tools for Design Assurance
  47. 47. Tools for Code Assurance
  48. 48. Tools for Code Assurance Assessment Items to choose Automated Tools for Code Assurance (1) User-Friendly • Usability • Analysis Report • Requirement to Evaluator (Expertise, Background Knowledge) (2) Effectiveness • Automation Level • Analysis Method • Detectable Vulnerability Type • Code Coverage • Licensing & Cost (3) Scalability • Supported Languages • Supported Platforms
  49. 49. CGC(Cyber Grand Challenge) Finalist • Mayhem CRS (ForAllSecure) • Xandra (TECHx) • Mechanical Phish (Shellphish) • Rebeus (Deep Red) • Crspy (Disekt) • Galactic (Codejitsu) • Jima (CSDS) Tools for Code Assurance
  50. 50. CGC (Cyber Grand Challenge) • CRS (Cyber Reasoning System) • Fully Automated Security Testing for Software (no human intervention!) Generate Input (Random, Mutation, Model-Based, … ) Input Generation Software Analysis & Excavate Vulnerability Vulnerability Scanning Crash is Exploitable? Crash Anaylsis Generate Exploit Code Automatically Exploit Generation Patched Binary Automatic Patching Tools for Code Assurance
  51. 51. Fortify SCA Assessment Items Description Usability GUI(Graphic User Interface), Easy to Use Analysis Report XML Report Requirement to Evaluator X Automation Level Fully Automated Analysis Method Static / Source Code Analyzer Detectable Vulnerability Type Hundreds of Vulnerability Code Coverage High Code Coverage Licensing & Cost Commercial (HP Enterprise) Supported Languages Java, .NET, C/C++, JSP, PL/SQL, TSQL, Javascript/Ajax, PHP, ASP, VB6, COBOL Supported Platforms Windows, Linux, Solaris, Mac OS X Tools for Code Assurance
  52. 52. CodeSonar Assessment Items Description Usability GUI, Easy to use Analysis Report HTML, XML, CSV Report Requirement to Evaluator X Automation Level Fully Automated Analysis Method Static / Source Code Analyzer / Binary Anaylzer Detectable Vulnerability Type Hundreds of Vulnerability Code Coverage High Code Coverage Licensing & Cost Commercial (Grammatech) Supported Languages C, C++, Java Supported Platforms Windows, Linux, Solaris Tools for Code Assurance
  53. 53. CheckMarx SAST Assessment Items Description Usability GUI, Easy to Use (Just throw the source code!) Analysis Report Dashboard Report (PDF, RTF, CSV, XML) Requirement to Evaluator X Automation Level Fully Automated Analysis Method Static / Source Code Analyzer Detectable Vulnerability Type Hundreds of Vulnerability Code Coverage High Code Coverage Licensing & Cost Commercial (CheckMarx) Supported Languages Java , Javascript , PHP , C# , VB.NET , VB6 , ASP.NET , C/C++ , Apex , Ruby , Perl , Objective-C , Python , Groovy , HTML5 , Swift , APEX , J2SE , J2EE Supported Platforms Android , iOS , Windows Tools for Code Assurance
  54. 54. KLEE Assessment Items. Description Usability CLI Analysis Report X Requirement to Evaluator O Automation Level Interactive Analysis Method Dynamic / Concolic Execution Detectable Vulnerability Type Memory Corruption Code Coverage High Code Coverage Licensing & Cost Non-Commercial (Researched by Stanford University) Supported Languages C, C++, Objective C Supported Platforms Linux Tools for Code Assurance
  55. 55. Mayhem (Research Paper Ver.) Assessment Items Description Usability CLI, Write Input Specification Analysis Report O (Exploit Type, Input Source, Symbolic Input Size, Precondition, Adivsory ,Exploit Generation Time) Requirement to Evaluator O Automation Level Interactive Analysis Method Dynamic / Concolic Execution Detectable Vulnerability Type Memory Corruption Code Coverage High Code Coverage Licensing & Cost Non-Commercial (Carnegie Mellon University) Supported Languages Raw Binary Code Supported Platforms Linux, Windows Tools for Code Assurance
  56. 56. SAGE Assessment Items Description Usability Unknown Analysis Report Unknown Requirement to Evaluator O Automation Level Interactive Analysis Method Dynamic / Whitebox Fuzz Testing Detectable Vulnerability Type Hundreds of Vulnerability Code Coverage Limited Code Coverage Licensing & Cost Restriced-Commercial (Microsoft) Supported Languages Raw Binary Code Supported Platforms Windows Tools for Code Assurance
  57. 57. Triton Assessment Items Description Usability CLI, Write Program based Triton Analysis Report X Requirement to Evaluator O Automation Level Interactive Analysis Method Dynamic / Concolic Execution / Framework Detectable Vulnerability Type Memory Corruption Code Coverage High Code Coverage Licensing & Cost Non-Commercial (Carnegie Mellon University) Supported Languages Raw Binary Code (Bordeaux University, Qarkslab) Supported Platforms Linux, Windows, Mac OS X Tools for Code Assurance
  58. 58. AFL (American Fuzzy Lop) Assessment Items Description Usability CLI(Command Line Interface) Install & Setup process is a little complexed. But provide colorful user interface and statistics. Analysis Report Crash/Vulnerability Type by Address Sanitizer Requirement to Evaluator O (Crash Analysis, Exploit Generation, Patching) Automation Level Interactive Analysis Method Dynamic / Guided Fuzz Testing Detectable Vulnerability Type Memory Corruption Code Coverage High Code Coverage (More time, More Coverage) Licensing & Cost Open Source (Michael Zalewski) Supported Languages C, C++, Objective C Supported Platforms Linux, *BSD, Solaris, Mac OS X On Linux, Only Binary(Blackbox) Testing Possible Tools for Code Assurance
  59. 59. IoTcube Assessment Items Description Usability Easy to Use (Web Interface, Drag & Drop) Analysis Report O Requirement to Evaluator X Automation Level Fully Automated Analysis Method Source Code Analysis (Code Clone Detection) Binary Fuzz Testing Network Vulnrability Testing (TLS) Detectable Vulnerability Type Hundreds of Vulnerability Code Coverage High Code Coverage Licensing & Cost Non-Commercial (CSSA, cssa.korea.ac.kr, iotcube.net) Supported Languages C/C++, Raw Binary Code Supported Platforms Linux, Windows, Mac OS X Tools for Code Assurance
  60. 60. Mechanical Phish (Shellphish CRS) Assessment Items Description Usability CLI, Install & Setup process is a little complexed but Easy to Use Analysis Report - Requirement to Evaluator X (Vulnerability Excavation, Crash Analysis, Exploit Generation, Patch) Automation Level Fully Automated Analysis Method Dynamic, Concolic Execution, Guided Fuzz Testing, Automatic Exploit Generation, Automatic Patching Detectable Vulnerability Type Memory Corruption Code Coverage High Code Coverage Licensing & Cost Non-Commercial (Shellphish) Supported Languages Raw Binary Code Supported Platforms Linux-Like Platforms(Custom by CGC), Intel x86 Tools for Code Assurance
  61. 61. Demo (Design / Code)
  62. 62. Conclusion
  63. 63. Conclusion There are many kind of Vulnerability Detection Tools developed by hackers, researchers. In present, we use these tools for security testing and evaluation. But there are some limits. • Objectivity • Coverage Recently, many of hackers research and develop automation tools that can find unknown vulnerability easily. We can’t apply these tools to security evaluation immediately. But if fully automated security testing techniques are developed and we make an effort to apply it for evaluation continuously, achieving high- assurance is not too far.
  64. 64. Acknowledgement This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIP) (R7117-16-0161,Anomaly detection framework for autonomous vehicles)
  65. 65. Q&A
  66. 66. Reference
  67. 67. Reference [1] Debra S. Herrmann, “A practical guide to Security Engineering and Information Assurance” [2] Sommerville, “Software Engineering, 9ed. 11 & 12, Dependability and Security Specification” [3] Charlie Miller, Chris Valasek, “Remote Exploitation of an Unaltered Passenger Vehicle” [4] Ross Anderson, “Security Engineering” [5] ISO/IEC/IEEE 15288 : 2015, “Systems and Software engineering-System life cycle process”, [6] Joe Jarzombek, “Software & Supply Chain Assurance : A Historical Perspective of Community Collaboration”, Homeland Security [7] David Burke, Joe Hurd and Aaron Tomb, “High Assurance Software Development”, 2010 [8] Ron Ross, Michael McEilley and Janet Carrier Oren, “NIST SP 800-160 : Systems Security Engineering – Consideration for a Multidisciplinary Approach in the Engineering of Trustworthy Secure Systems”, 2016 [9] Scott A.Lintelman, Krishna Sampigethaya, Mingyan Li, Radha Poovendran, Richard V. Robinson, “High Assurance Aerospace CPS & Implications for the Automotive Industry”, 2015 [10] NIAP, “Common Criteria-Evaluation and Validation Scheme, Publication #3, Guidance to Validators version 3”, 2014 [11] ISO/IEC 27034-2, “Information technology – Security techniques – Application Security”, 2015 [12] Paul R. Croll, “ISO/IEC/IEEE 15026, Systems and Software Assurance”, 21st Annual Systems and Software Technology Conference, 2009
  68. 68. Reference [13] EURO-MILS, “Secure European Virtualisation for Trustworthy Applications in Critical Domains, Used Formal Methods”, 2015 [14] Vijay D’Silva, Daniel Kroening, and Georg Weissenbacher, “A Survey of Automated Techniques for Formal Software Verification”, 2008 [15] Daniel Potts, Rene Bourquin, Lesile Andresen, “Mathematically Verified Software Kernals: Rasing the Bar for High Assurance Implementation [16] Bernhard Beckert, Daniel Bruns, Sarah Grebing, “Mind the Gap : Formal Verification and the Common Criteria“, 2010 [17] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolankski, Michel Norrich, Thomas Sewell, Harvey Tuch, Simon Winwood, “seL4 : Formal Verification of an OS Kernel”, 2009 [18] Gerwin Klein, NICTA, “Operating System Verification – An Overview”, 2009 [19] Jesus Diaz, David Arroyo, Francisco B. Rodriguez, “A formal methodology for integral security design and verification of network protocols”, 2012 [20] Yoshikazu Hanatanil, Miyako Ohkubo, Sinichiro Matsuo, Kazuo Sakiyama, and Kazuo Ohta, “A Study on Computational Formal Verification for Practical Cryptographic Protocol: The Case of Synchronous RFID Authentication”, 2011 [21] Alexandre Melo Braga, Ricardo Hahab, “A Survey on Tools and Techniques for the Programming and Verification of Secure Cryptographic Software”, 2015
  69. 69. Reference [22] Shinichiro Matsuo, Kunihiko Miyazaki, Akira Otsuka, David Basin, “How to Evaluate the Security of Real-life Cryptographic Protocol? The cases of ISO/IEC 29128 and CRYPTREC, 2010 [23] Bruno Blanchet, Ben Smyth, and Vincent Cheval, “ProVerif 1.94pl1: Automatic Cryptographic Protocol Verifier, User Manual and Tutorial”, 2016 [24] Charles B. Weinstock, John B. Goodennough, “Toward an Assurance Case Practice for Medical Devices”, 2009 [25] CISCO, “Building Trustworthy Systems with Cisco Secure Development Lifecycle”, 2016 [26] Yannick Moy, Emmanuel Ledinot, Herve Delseny, Virginie Wiels, Benjamin Monte, “Testing or Formal Verification : DC-178C Alternatives and Industrial Experience”, 2013 [27] Karen Scarfone, Murugiah Souppaya, Amanda Cody, Angela Orebaugh, “NIST SP 800-115, Technical Guide to Information Security Testing and Assessment – Recommandations of the National Institue of Standards and Technology”, 2008 [28] Steve Lipner, Microsoft, “The Security Development Lifecycle”, 2010 [29] Michael Felderer, Ruth Breu, Matthias Buchler, “Security Testing : A Survey”, 2016 [30] Vijay D’Silva, Daniel Kroening, George Weissenbacher, “A Survey of Automated Techniques for Formal Software Verification” [31] John Rushby, Xidong Xu, Rangarajan and Thomas L. Weaver, “Understanding and Evaluating Assurance Case”, 2015 [32] David J.Rinehart, John C. Knight, Jonathan Rowanhill, “Current Practices in Constructing and Evaluating Assurance Case with Application to Aviation”, 2015 [33] The Government of Japan, “Cybersecurity Strategy 2015”
  70. 70. Reference [34] Yasu Taniwaki, Deputy Director-General National Information Security Center, “Cybersecurity Strategy in Japan”, 2014 [35] “The NRL Protocol Analyzer : An Overview”, 1994 [36] Bruno Blanchet, “Automatic Verification of security protocols : the tools ProVerif and CryptoVerif”, 2011 [37] Tobias Nipkow, “Programming and Proving in Isabelle/HOL”, 2016 [38] Assistant Secretary of the Navy Chief System Engineer, “Software Security Assessment Tools Review”, 2009 [39] S.Santiago, C.Talcott, S.Escobar, C.Meadows, J.Meseguer, “A Graphical User Interface for Maude-NPA”, 2009 [40] NIST, "Source Code Security Analyzers" [41] Cadar, Cristian, "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs", 2008 [42] Cha, Sang Kil, "Unleashing MAYHEM on Binary Code", 2012 [43] Giovanni Vigna, "Autonomous Hacking: The New Frontiers of Attack and Defense", 2016 [44] Antonio Bianchi, "A Dozen Years of Shellphish From DEFCON to the Cyber Grand Challenge", 2015 [45] Jonathan Salwan, "Triton: Concolic Execution Framework", 2016 [46] Godefroid, "SAGE: Whitebox Fuzzing for Security Testing", 2012 [47] Michael Zalewski, "American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/)", 2015 [48] Vegard Nossum, Oracle, "Filesystem Fuzzing with American Fuzzy Lop", 2016 [49] Hongzhe Li, "CLORIFI: software vulnerability discovery using code clone verification", 2015 [50] Stephens, "Driller: Augmenting Fuzzing Through Selective Symbolic Execution", 2016 [51] John Rushby, “The Interpretation and Evaluation of Assurance Cases”, SRI International Technical Report, 2015

×