In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, he shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, attendees get to practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.
Expert tester and speaker JeanAnn Harrison reveals critical mobile tests that can be used on many mobile devices, even proprietary. Exercises and ideas for testing that you can use and build on. Sponsored by XBOSoft.
This document provides an overview and learning plan for a course on secure programming. It discusses key concepts like understanding security as a mindset, process, risk management approach, and multidisciplinary science. Specific topics covered include security definitions, vulnerability databases, secure software engineering, security assessment/testing, and understanding the costs of patching insecure software.
CompTIA exam study guide presentations by instructor Brian Ferrill, PACE-IT (Progressive, Accelerated Certifications for Employment in Information Technology)
"Funded by the Department of Labor, Employment and Training Administration, Grant #TC-23745-12-60-A-53"
Learn more about the PACE-IT Online program: www.edcc.edu/pace-it
This document provides an agenda and overview for conducting a comprehensive physical security risk assessment. It includes definitions of physical security, outlines roles and responsibilities, and provides sample tools and checklists to guide the assessment. When to conduct an assessment, why it's important, and how to develop a risk appetite and project plan are also covered. The goal is to identify vulnerabilities and risks in order to create an effective corrective action plan to improve security.
The impact of innovation on travel and tourism industries (World Travel Marke...Brian Solis
From the impact of Pokemon Go on Silicon Valley to artificial intelligence, futurist Brian Solis talks to Mathew Parsons of World Travel Market about the future of travel, tourism and hospitality.
We’re all trying to find that idea or spark that will turn a good project into a great project. Creativity plays a huge role in the outcome of our work. Harnessing the power of collaboration and open source, we can make great strides towards excellence. Not just for designers, this talk can be applicable to many different roles – even development. In this talk, Seasoned Creative Director Sara Cannon is going to share some secrets about creative methodology, collaboration, and the strong role that open source can play in our work.
Reuters: Pictures of the Year 2016 (Part 2)maditabalnco
This document contains 20 photos from news events around the world between January and November 2016. The photos show international events like the US presidential election, the conflict in Ukraine, the migrant crisis in Europe, the Rio Olympics, and more. They also depict human interest stories and natural phenomena from various countries.
Expert tester and speaker JeanAnn Harrison reveals critical mobile tests that can be used on many mobile devices, even proprietary. Exercises and ideas for testing that you can use and build on. Sponsored by XBOSoft.
This document provides an overview and learning plan for a course on secure programming. It discusses key concepts like understanding security as a mindset, process, risk management approach, and multidisciplinary science. Specific topics covered include security definitions, vulnerability databases, secure software engineering, security assessment/testing, and understanding the costs of patching insecure software.
CompTIA exam study guide presentations by instructor Brian Ferrill, PACE-IT (Progressive, Accelerated Certifications for Employment in Information Technology)
"Funded by the Department of Labor, Employment and Training Administration, Grant #TC-23745-12-60-A-53"
Learn more about the PACE-IT Online program: www.edcc.edu/pace-it
This document provides an agenda and overview for conducting a comprehensive physical security risk assessment. It includes definitions of physical security, outlines roles and responsibilities, and provides sample tools and checklists to guide the assessment. When to conduct an assessment, why it's important, and how to develop a risk appetite and project plan are also covered. The goal is to identify vulnerabilities and risks in order to create an effective corrective action plan to improve security.
The impact of innovation on travel and tourism industries (World Travel Marke...Brian Solis
From the impact of Pokemon Go on Silicon Valley to artificial intelligence, futurist Brian Solis talks to Mathew Parsons of World Travel Market about the future of travel, tourism and hospitality.
We’re all trying to find that idea or spark that will turn a good project into a great project. Creativity plays a huge role in the outcome of our work. Harnessing the power of collaboration and open source, we can make great strides towards excellence. Not just for designers, this talk can be applicable to many different roles – even development. In this talk, Seasoned Creative Director Sara Cannon is going to share some secrets about creative methodology, collaboration, and the strong role that open source can play in our work.
Reuters: Pictures of the Year 2016 (Part 2)maditabalnco
This document contains 20 photos from news events around the world between January and November 2016. The photos show international events like the US presidential election, the conflict in Ukraine, the migrant crisis in Europe, the Rio Olympics, and more. They also depict human interest stories and natural phenomena from various countries.
In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, Jon shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.
Software Testing Attacks for Mobile and Embedded DevicesXBOSoft
Jon Hagar author of "Software Test Attacks to Break Mobile and Embedded Devices" presents Software testing concepts for mobile and imbedded devices in this webinar- hosted by XBOSoft.
Mobile App Testing: The Good, the Bad, and the UglyTechWell
The document discusses mobile application testing and provides examples of issues that can arise. It describes current problems like management not allowing enough testing time or prioritizing speed over quality. This can lead to bugs being missed and poor user experiences. The document also introduces taxonomies to help categorize different types of bugs, like those related to timing, to aid in more effective testing. Overall, it advocates for improved mobile app testing practices to avoid common pitfalls and ensure high quality user experiences.
XBOSoft Mobile Security Webinar with Jon D. HagarXBOSoft
Mobile Security is a real world threat in today's technology sector, these slides explore testing attack concepts and how to prevent hacks and vulnerabilities creeping up in your mobile app development or device deployment. Jon D Hagar goes through methodologies all software developers and software testers need to follow to ensure mobile security risks are minimizes and controlled.
IoT Software Testing Challenges: The IoT World Is Really DifferentTechWell
The Internet of Things (IoT) is poised to become the next growth area—and biggest challenge—for software development and testing. Although many traditional test techniques and strategies remain viable, IoT testing includes working with huge amounts of data, multiple communication channels, device protocols, resource limitations (battery or memory), sensors, controllers, cloud-hardware-device integration, and security concerns. Jon Hagar says that successful IoT testers must develop new knowledge and skills and apply them based on real data and proven test design methods. Testing analytics should include raw test data, data relationships across software integration boundaries, and social media inputs—as well as a keen understanding of sociological and psychological factors. Jon shares insights into math-based testing, model-based testing, attack-based and exploratory testing for IoT applications and systems. Take back a new holistic view for your IoT testing which considers the world environment, connected systems, local systems, and the IoT device itself.
Use Combinatorial Testing for Mobile Device FragmentationJosiah Renaudin
A common problem in mobile systems testing is the number of hardware, operational, and software configurations that need to be tested. For example, the so-called Android fragmentation problem might lead a test team to test hundreds of device and software configurations, yielding thousands or even tens-of-thousands of tests. A branch of mathematics, called combinatorics, and associated tools exist that allow teams to minimize the number of test cases required, while assuring high error finding percentages. Jon Hagar defines the fragmentation problem and then examines test patterns supported by tools that can help improve testing success. Jon outlines how combinatorial test patterns can be applied to other testing situations. To solve real-world fragmentation problems, he identifies specific tools, which you can take back to your project for quick use. Reference work and data are provided to help your team justify adding combinatorial testing to your mobile test activities.
Implement Combinatorial Test Patterns for Better Mobile and IoT TestingJosiah Renaudin
A common problem in mobile and IoT systems is the large number and combinations of hardware, operational, and software configurations that need to be tested. For example, the so-called Android fragmentation problem might lead a test team to test hundreds of device and several software configurations, potentially yielding thousands or even tens of thousands of tests. Combinatorial testing, a technique involving mathematics and specific tooling, allows teams to reduce the number of test cases, while still assuring good error finding capabilities. Jon Hagar examines test combinatorial patterns supported by tools that will help you speed up testing these many configurations and use for other test tasks, too. During this session Jon will identify and demonstrate specific tools to solve real-world mobile and IoT testing problems. Take back reference materials and data to help your team justify adding combinatorial testing to its toolkit and regular testing activities.
Exploratory testing and the mobile tester : A presentation by Jon HagarGallop Solutions
The document discusses various testing attacks that can be performed on mobile and embedded systems. It begins by providing context on mobile testing and defines key terms. It then outlines 33 different attacks that target common issues like static code analysis bugs, data computation errors, hardware/software interface problems, security vulnerabilities, and usability defects. Several attacks are described in more detail, including developer attacks, basic tester attacks, and dangerous security attacks. The document aims to educate testers on effective exploratory testing techniques for breaking mobile and IoT devices.
Software Attacks for Embedded, Mobile, and Internet of ThingsTechWell
In the world of embedded systems, mission-critical mobile apps, and the Internet of Things (IoT), developers and testers must do more than just look for feature bugs. To find potential failures and serious security errors, their arsenal should include attack-based exploratory testing. In the tradition of James Whittaker’s How to Break Software books, Jon Hagar applies the “attack” concept to embedded, mobile, and IoT software. Jon examines common industry patterns of product failures and shares a set of his favorite software test attacks for native, web-based, and hybrid apps. He explains when and how to conduct the attacks, including the pros and cons of some attacks. Take back an arsenal of at least three basic tester attacks, three developer attacks, and three security attacks that you can employ on your current or next project.
Mobile App Testing: Design Automation Patterns You Should UseTechWell
In mobile app development, better test design is important to project velocity and user satisfaction. Jon Hagar explores underused or poorly practiced test design automation approaches that you should employ in development and testing. Jon begins by defining the domain of mobile app software and examines common industry patterns of product failures. He then shares three approaches you can use to speed development and improve quality for native, web-based, and hybrid apps. The methods examined—each supported with detailed checklists—are combinatorial testing, model-based testing, and user experience testing. Jon explains when, where, and how each testing approach can be used to support improved testing and to benefit the whole team. In addition to mobile apps, you and your team can use these same three approaches in other software environments to reduce technical debt during development.
IoT Software Testing Challenges: The IoT World Is Really DifferentTechWell
With billions of devices containing new software connected to the Internet, the Internet of Things (IoT) is poised to become the next growth area for software development and testing. Although many traditional test techniques and strategies remain viable, challenges in IoT testing include huge amounts of data, multiple communication channels, device protocols, resource limitations (battery or memory), addressing sensors and controllers, cloud-hardware-device integration, and security concerns. Jon Hagar says that for IoT testers to be successful, they must develop new knowledge and skills, and apply them based on real data and proven test design methods. Testing analytics should include raw test data, data relationships across software integration boundaries, and social media inputs, as well as a keen understanding of sociological and psychological factors. Jon shares insights into math-based testing, model-based testing, attack-based exploratory testing, and appropriate types of standards as basics of IoT testing. Take back a new holistic view for your IoT testing which considers the world environment, connected systems, local systems, and the IoT device itself.
Apply Problem Solving Techniques to Routine Malfunctions.pptxwesendesta2
The document provides information on identifying and determining the root causes of routine computer problems. It discusses identifying problems as hardware, software, user, or procedural related. Common hardware issues include problems with components like the power supply, motherboard, or CPU. Software problems can arise from operating systems, applications, or outdated drivers. User errors and procedural problems are also examined. The document outlines a six-step problem solving model to determine the fundamental root causes of identified issues through investigation and developing potential cause statements. Common computer problems and their potential solutions are also reviewed.
Difference between hardware and software computer hardware vs softwareSwapan Das
This document discusses the difference between computer hardware and software. Computer hardware refers to the physical devices that can be seen and touched, such as a monitor, keyboard, mouse, and storage devices. Computer software refers to sets of instructions and code that tell the computer hardware how to perform tasks; software cannot usually be seen or touched. Some key differences are that hardware can fail over time due to wear, while software failures tend to be due to bugs and can be fixed; hardware is physical while software is logical; and hardware is used to run software programs.
This document provides an overview of the Secure Software Development Lifecycle (SSDLC). It discusses how SSDLC differs from traditional development by focusing on security requirements, design, testing, and operations. Key aspects include threat modeling to identify risks, the principle of least privilege, extensive testing and logging, and having policies and response plans for security incidents. The goal of SSDLC is to build resilience, stability, and trust into software through a more proactive and defensive approach throughout the entire development lifecycle.
The document discusses various types and levels of software testing. It defines software testing as analyzing a software item to detect differences between existing and required conditions (i.e. defects). The key types discussed are positive and negative testing, white-box and black-box testing. The levels covered are unit testing, integration testing, system testing, and acceptance testing. Various testing tools are also listed for different testing purposes such as source code testing, functional testing, performance testing, and database testing.
The document discusses techniques for testing software security, as traditional testing methods are not well-suited for finding security bugs. It outlines several approaches for identifying unintended side effects, including monitoring for unexpected interactions with the environment, injecting faults to test error handling, and attacking dependencies and implementations. Specifically, the document recommends testing applications' use of resources like files, memory, and network availability under stressful conditions to identify potential vulnerabilities.
Gopal Kumar presented on software quality assurance and testing done during an industrial training. The presentation covered:
1) An overview of the organization SpiceRetail Ltd and their mobile phone offerings.
2) Different types and levels of testing including manual, automation, black box, white box and functional/non-functional testing.
3) Challenges in mobile device testing like supporting different targets and operating systems.
4) A proposed interruption matrix approach to help select meaningful test cases accounting for device interruptions and states.
Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrisonXBOSoft
To Automate or not to Automate your Mobile Testing.
In mobile testing just poking at the GUI will leave bugs hiding. So different tests and a variety of testers are needed. Context is also important; there is no one test set or test approach that will work all the time.
JeanAnn Harrison has years of experience with mobile testing and is a well-known figure in the QA and software testing community. She regularly speaks at conferences and publishes in software testing magazines.
In these slides JeanAnn discusses mobile testing strategies that deliver the right results.
You will learn:
- Types of Mobile Testing
- When and when not to automate your mobile testing
- Mobile exploratory testing strategies and guidelines
- Lesson learned
We are all aware of the current risks when developing a connected product, especially with vehicles since much is at stake both from an information and safety perspective. In this workshop, we will learn how to build Security requirements, architect, design, test and produce Safety and Security critical components using a methodology that works in harmony both with Engineering and Security
Isabel Evans stopped drawing and painting after being told she was not very good at it, which led to a loss of confidence in her creative and professional abilities. However, she realized that attempting creative activities is important for cognitive and emotional development, and that making mistakes and learning from failures allows for growth. By reengaging with failure through art and with support from others, Isabel was able to regain confidence in her abilities and reboot her career. The document discusses different perspectives on failure and the importance of learning from mistakes.
Instill a DevOps Testing Culture in Your Team and Organization TechWell
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
More Related Content
Similar to How to Break Software: Embedded Edition
In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, Jon shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.
Software Testing Attacks for Mobile and Embedded DevicesXBOSoft
Jon Hagar author of "Software Test Attacks to Break Mobile and Embedded Devices" presents Software testing concepts for mobile and imbedded devices in this webinar- hosted by XBOSoft.
Mobile App Testing: The Good, the Bad, and the UglyTechWell
The document discusses mobile application testing and provides examples of issues that can arise. It describes current problems like management not allowing enough testing time or prioritizing speed over quality. This can lead to bugs being missed and poor user experiences. The document also introduces taxonomies to help categorize different types of bugs, like those related to timing, to aid in more effective testing. Overall, it advocates for improved mobile app testing practices to avoid common pitfalls and ensure high quality user experiences.
XBOSoft Mobile Security Webinar with Jon D. HagarXBOSoft
Mobile Security is a real world threat in today's technology sector, these slides explore testing attack concepts and how to prevent hacks and vulnerabilities creeping up in your mobile app development or device deployment. Jon D Hagar goes through methodologies all software developers and software testers need to follow to ensure mobile security risks are minimizes and controlled.
IoT Software Testing Challenges: The IoT World Is Really DifferentTechWell
The Internet of Things (IoT) is poised to become the next growth area—and biggest challenge—for software development and testing. Although many traditional test techniques and strategies remain viable, IoT testing includes working with huge amounts of data, multiple communication channels, device protocols, resource limitations (battery or memory), sensors, controllers, cloud-hardware-device integration, and security concerns. Jon Hagar says that successful IoT testers must develop new knowledge and skills and apply them based on real data and proven test design methods. Testing analytics should include raw test data, data relationships across software integration boundaries, and social media inputs—as well as a keen understanding of sociological and psychological factors. Jon shares insights into math-based testing, model-based testing, attack-based and exploratory testing for IoT applications and systems. Take back a new holistic view for your IoT testing which considers the world environment, connected systems, local systems, and the IoT device itself.
Use Combinatorial Testing for Mobile Device FragmentationJosiah Renaudin
A common problem in mobile systems testing is the number of hardware, operational, and software configurations that need to be tested. For example, the so-called Android fragmentation problem might lead a test team to test hundreds of device and software configurations, yielding thousands or even tens-of-thousands of tests. A branch of mathematics, called combinatorics, and associated tools exist that allow teams to minimize the number of test cases required, while assuring high error finding percentages. Jon Hagar defines the fragmentation problem and then examines test patterns supported by tools that can help improve testing success. Jon outlines how combinatorial test patterns can be applied to other testing situations. To solve real-world fragmentation problems, he identifies specific tools, which you can take back to your project for quick use. Reference work and data are provided to help your team justify adding combinatorial testing to your mobile test activities.
Implement Combinatorial Test Patterns for Better Mobile and IoT TestingJosiah Renaudin
A common problem in mobile and IoT systems is the large number and combinations of hardware, operational, and software configurations that need to be tested. For example, the so-called Android fragmentation problem might lead a test team to test hundreds of device and several software configurations, potentially yielding thousands or even tens of thousands of tests. Combinatorial testing, a technique involving mathematics and specific tooling, allows teams to reduce the number of test cases, while still assuring good error finding capabilities. Jon Hagar examines test combinatorial patterns supported by tools that will help you speed up testing these many configurations and use for other test tasks, too. During this session Jon will identify and demonstrate specific tools to solve real-world mobile and IoT testing problems. Take back reference materials and data to help your team justify adding combinatorial testing to its toolkit and regular testing activities.
Exploratory testing and the mobile tester : A presentation by Jon HagarGallop Solutions
The document discusses various testing attacks that can be performed on mobile and embedded systems. It begins by providing context on mobile testing and defines key terms. It then outlines 33 different attacks that target common issues like static code analysis bugs, data computation errors, hardware/software interface problems, security vulnerabilities, and usability defects. Several attacks are described in more detail, including developer attacks, basic tester attacks, and dangerous security attacks. The document aims to educate testers on effective exploratory testing techniques for breaking mobile and IoT devices.
Software Attacks for Embedded, Mobile, and Internet of ThingsTechWell
In the world of embedded systems, mission-critical mobile apps, and the Internet of Things (IoT), developers and testers must do more than just look for feature bugs. To find potential failures and serious security errors, their arsenal should include attack-based exploratory testing. In the tradition of James Whittaker’s How to Break Software books, Jon Hagar applies the “attack” concept to embedded, mobile, and IoT software. Jon examines common industry patterns of product failures and shares a set of his favorite software test attacks for native, web-based, and hybrid apps. He explains when and how to conduct the attacks, including the pros and cons of some attacks. Take back an arsenal of at least three basic tester attacks, three developer attacks, and three security attacks that you can employ on your current or next project.
Mobile App Testing: Design Automation Patterns You Should UseTechWell
In mobile app development, better test design is important to project velocity and user satisfaction. Jon Hagar explores underused or poorly practiced test design automation approaches that you should employ in development and testing. Jon begins by defining the domain of mobile app software and examines common industry patterns of product failures. He then shares three approaches you can use to speed development and improve quality for native, web-based, and hybrid apps. The methods examined—each supported with detailed checklists—are combinatorial testing, model-based testing, and user experience testing. Jon explains when, where, and how each testing approach can be used to support improved testing and to benefit the whole team. In addition to mobile apps, you and your team can use these same three approaches in other software environments to reduce technical debt during development.
IoT Software Testing Challenges: The IoT World Is Really DifferentTechWell
With billions of devices containing new software connected to the Internet, the Internet of Things (IoT) is poised to become the next growth area for software development and testing. Although many traditional test techniques and strategies remain viable, challenges in IoT testing include huge amounts of data, multiple communication channels, device protocols, resource limitations (battery or memory), addressing sensors and controllers, cloud-hardware-device integration, and security concerns. Jon Hagar says that for IoT testers to be successful, they must develop new knowledge and skills, and apply them based on real data and proven test design methods. Testing analytics should include raw test data, data relationships across software integration boundaries, and social media inputs, as well as a keen understanding of sociological and psychological factors. Jon shares insights into math-based testing, model-based testing, attack-based exploratory testing, and appropriate types of standards as basics of IoT testing. Take back a new holistic view for your IoT testing which considers the world environment, connected systems, local systems, and the IoT device itself.
Apply Problem Solving Techniques to Routine Malfunctions.pptxwesendesta2
The document provides information on identifying and determining the root causes of routine computer problems. It discusses identifying problems as hardware, software, user, or procedural related. Common hardware issues include problems with components like the power supply, motherboard, or CPU. Software problems can arise from operating systems, applications, or outdated drivers. User errors and procedural problems are also examined. The document outlines a six-step problem solving model to determine the fundamental root causes of identified issues through investigation and developing potential cause statements. Common computer problems and their potential solutions are also reviewed.
Difference between hardware and software computer hardware vs softwareSwapan Das
This document discusses the difference between computer hardware and software. Computer hardware refers to the physical devices that can be seen and touched, such as a monitor, keyboard, mouse, and storage devices. Computer software refers to sets of instructions and code that tell the computer hardware how to perform tasks; software cannot usually be seen or touched. Some key differences are that hardware can fail over time due to wear, while software failures tend to be due to bugs and can be fixed; hardware is physical while software is logical; and hardware is used to run software programs.
This document provides an overview of the Secure Software Development Lifecycle (SSDLC). It discusses how SSDLC differs from traditional development by focusing on security requirements, design, testing, and operations. Key aspects include threat modeling to identify risks, the principle of least privilege, extensive testing and logging, and having policies and response plans for security incidents. The goal of SSDLC is to build resilience, stability, and trust into software through a more proactive and defensive approach throughout the entire development lifecycle.
The document discusses various types and levels of software testing. It defines software testing as analyzing a software item to detect differences between existing and required conditions (i.e. defects). The key types discussed are positive and negative testing, white-box and black-box testing. The levels covered are unit testing, integration testing, system testing, and acceptance testing. Various testing tools are also listed for different testing purposes such as source code testing, functional testing, performance testing, and database testing.
The document discusses techniques for testing software security, as traditional testing methods are not well-suited for finding security bugs. It outlines several approaches for identifying unintended side effects, including monitoring for unexpected interactions with the environment, injecting faults to test error handling, and attacking dependencies and implementations. Specifically, the document recommends testing applications' use of resources like files, memory, and network availability under stressful conditions to identify potential vulnerabilities.
Gopal Kumar presented on software quality assurance and testing done during an industrial training. The presentation covered:
1) An overview of the organization SpiceRetail Ltd and their mobile phone offerings.
2) Different types and levels of testing including manual, automation, black box, white box and functional/non-functional testing.
3) Challenges in mobile device testing like supporting different targets and operating systems.
4) A proposed interruption matrix approach to help select meaningful test cases accounting for device interruptions and states.
Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrisonXBOSoft
To Automate or not to Automate your Mobile Testing.
In mobile testing just poking at the GUI will leave bugs hiding. So different tests and a variety of testers are needed. Context is also important; there is no one test set or test approach that will work all the time.
JeanAnn Harrison has years of experience with mobile testing and is a well-known figure in the QA and software testing community. She regularly speaks at conferences and publishes in software testing magazines.
In these slides JeanAnn discusses mobile testing strategies that deliver the right results.
You will learn:
- Types of Mobile Testing
- When and when not to automate your mobile testing
- Mobile exploratory testing strategies and guidelines
- Lesson learned
We are all aware of the current risks when developing a connected product, especially with vehicles since much is at stake both from an information and safety perspective. In this workshop, we will learn how to build Security requirements, architect, design, test and produce Safety and Security critical components using a methodology that works in harmony both with Engineering and Security
Similar to How to Break Software: Embedded Edition (20)
Isabel Evans stopped drawing and painting after being told she was not very good at it, which led to a loss of confidence in her creative and professional abilities. However, she realized that attempting creative activities is important for cognitive and emotional development, and that making mistakes and learning from failures allows for growth. By reengaging with failure through art and with support from others, Isabel was able to regain confidence in her abilities and reboot her career. The document discusses different perspectives on failure and the importance of learning from mistakes.
Instill a DevOps Testing Culture in Your Team and Organization TechWell
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
Test Design for Fully Automated Build ArchitectureTechWell
This document summarizes a half-day tutorial on test design for fully automated build architectures presented by Melissa Benua of mParticle at STAREAST 2018. The tutorial covered guiding principles for test design including prioritizing important and reliable tests, structuring automated pipelines around components, packages, and releases, and monitoring test results through code coverage, flaky test handling, and logging versus counters. It also included exercises mapping test cases to functional boundaries and categories of tests to pipeline stages.
System-Level Test Automation: Ensuring a Good StartTechWell
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
Build Your Mobile App Quality and Test StrategyTechWell
Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.
Testing Transformation: The Art and Science for SuccessTechWell
Technologies, testing processes, and the role of the tester have evolved significantly in the past few years with the advent of agile, DevOps, and other new technologies. It is critical that we testing professionals evaluate ourselves and continue to add tangible value to our organizations. In your work, are you focused on the trivial or on real game changers? Jennifer Bonine describes critical elements that help you artfully blend people, process, and technology to create a synergistic relationship that adds value. Jennifer shares ideas on mastering politics, maneuvering core vs. context, and innovating your technology strategies and processes. She explores how new processes can be introduced in an organization, what the role of organizational culture is in determining the success of a project, and how you can know what tools will add value vs. simply adding overhead and complexity. Jennifer reviews critically needed tester skills and discusses a continual learning model to evolve your skills and stay relevant. This discussion can lead you to technologies, processes, and skills you can stake your career on.
We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
Many teams go crazy because of brittle, high-maintenance automated test suites. Jim Holmes helps you understand how to create a flexible, maintainable, high-value suite of functional tests using Selenium WebDriver. Learn the basics of what to test, what not to test, and how to avoid overlapping with other types of testing. Jim includes both philosophical concepts and hands-on coding. Testers who haven't written code should not be intimidated! We'll pair you up to make sure you're successful. Learn to create practical tests dealing with advanced situations such as input validation, AJAX delays, and working with file downloads. Additionally, discover when you need to work together with developers to create a system that's more easily testable. This tutorial focuses primarily on automating web tests, but many of the same concepts can be applied to other UI environments. Demos and labs will be in C# and Java using WebDriver. Leave this tutorial having learned how to write high-value WebDriver tests—and stay sane while doing so.
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
Chris Parlette maintains that renting infrastructure on demand is the most disruptive trend in IT in decades. In 2016, enterprises spent $23B on public cloud IaaS services. By 2020, that figure is expected to reach $65B. The public cloud is now used like a utility, and like any utility, there is waste. Who's responsible for optimizing the infrastructure and reducing wasted expenses? It’s DevOps. The excess expense, known as cloud waste, comprises several interrelated problems: services running when they don't need to be, improperly sized infrastructure, orphaned resources, and shadow IT. There are a few core tenets of DevOps—holistic thinking, no silos, rapid useful feedback, and automation—that can be applied to reducing your cloud waste. Join Chris to learn why you should include continuous cost optimization in your DevOps processes. Automate cost control, reduce your cloud expenses, and make your life easier.
Transform Test Organizations for the New World of DevOpsTechWell
With the recent emergence of DevOps across the industry, testing organizations are being challenged to transform themselves significantly within a short period of time to stay meaningful within their organizations. It’s not easy to plan and approach these changes considering the way testing organizations have remained structured for ages. These challenges start from foundational organizational structures and can cut across leadership influence, competencies, tools strategy, infrastructure, and other dimensions. Sumit Kumar shares his experience assisting various organizations to overcome these challenges using an organized DevOps enablement framework. The framework includes radical restructuring, turning the tools strategy upside down, a multidimensional workforce enablement supported by infrastructure changes, redeveloped collaborations models, and more. From his real world experiences Sumit shares tips for approaching this journey and explains the roadmap for testing organizations to transform themselves to lead the quality in DevOps.
The Fourth Constraint in Project Delivery—LeadershipTechWell
All too often, the triple constraints—time, cost, and quality—are bandied about as if they are the be-all, end-all. While they are important, leadership—the fourth and larger underpinning constraint—influences the first three. Statistics on project success and failure abound, and these measurements are usually taken against the triple constraints. According to the Project Management Institute, only 53 percent of projects are completed within budget, and only 49 percent are completed on time. If so many projects overrun budget and are late, we can’t really say, “Good, fast, or cheap—pick two.” Rob Burkett talks about leadership at every level of a team. He shares his insights and stories gleaned from his years of IT and project management experience. Rob speaks to some of the glaring difficulties in the workplace in general and some specifically related to IT delivery and project management. Leave with a clearer understanding of how to communicate with teams and team members, and gain a better understanding of how you can be a leader—up and down your organization.
Resolve the Contradiction of Specialists within Agile TeamsTechWell
As teams grow, organizations often draw a distinction between feature teams, which deliver the visible business value to the user, and component teams, which manage shared work. Steve Berczuk says that this distinction can help organizations be more productive and scale effectively, but he recognizes that not all shared work fits into this model. Some work is best handled by “specialists,” that is people with unique skills. Although teams composed entirely of T-shaped people is ideal, certain skills are hard to come by and are used irregularly across an organization. Since these specialists often need to work closely with teams, rather than working from their own backlog, they don’t fit into the component team model. The use of shared resources presents challenges to the agile planning model. Steve Berczuk shares how teams such as those providing infrastructure services and specialists can fit into a feature+component team model, and how variations such as embedding specialists in a scrum team can both present process challenges and add significant value to both the team and the larger organization.
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
Metrics don’t have to be a necessary evil. If done right, metrics can help guide us to make better forward-looking decisions, rather than being used for simply managing or monitoring. They can help us identify trade-offs between options for what to do next versus punitive or worse, purely managerial measures. Steve Martin won’t be giving the Top Ten List of field-tested metrics you should use. Instead, in this interactive mini-workshop, he leads you through the critical thinking necessary for you to determine what is right for you to measure. First, Steve explores why you want to measure something—whether it’s for a team, a portfolio, or even an agile transformation. Next, he provides multiple real-life metrics examples to help drive home concepts behind characteristics of good and bad metrics. Finally, Steve shows how to run his field-tested agile game—Pin the Tail on the Metric. Take back this activity to help you guide metrics conversations at your organization.
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
A hierarchy is an organizational network that has a top and a bottom, and where position is determined by rank, importance, and value. A holarchy is a network that has no top or bottom and where each person’s value derives from his ability, rather than position. As more companies seek the benefits of agile, leaders need to build and sustain delivery capability while scaling agile without introducing unnecessary process and overhead. The Agile Performance Holarchy (APH) is an empirical model for scaling and sustaining agility while continuing to deliver great products. Jeff Dalton designed the APH by drawing from lessons learned observing and assessing hundreds of agile companies and teams. The APH helps implement a holarchy—a system composed of interacting organizational units called holons—centered on a series of performance circles that embody the behaviors of high performing agile organizations. Jeff describes how APH provides guidelines in the areas of leadership, values, teaming, visioning, governing, building, supporting, and engaging within an all-agile organization. Join Jeff to see what the APH is all about and how you can use it in your team and organization.
A Business-First Approach to DevOps ImplementationTechWell
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Databases in a Continuous Integration/Delivery ProcessTechWell
The document summarizes a presentation about including databases in a continuous integration/delivery process. It discusses treating database code like application code by placing it under version control and integrating databases into the DevOps software development pipeline. This allows databases to be built, tested, and released like other software through continuous integration, delivery, and deployment.
Mobile Testing: What—and What Not—to AutomateTechWell
Organizations are moving rapidly into mobile technology, which has significantly increased the demand for testing of mobile applications. David Dangs says testers naturally are turning to automation to help ease the workload, increase potential test coverage, and improve testing efficiency. But should you try to automate all things mobile? Unfortunately, the answer is not always clear. Mobile has its own set of complications, compounded by a wide variety of devices and OS platforms. Join David to learn what mobile testing activities are ripe for automation—and those items best left to manual efforts. He describes the various considerations for automating each type of mobile application: mobile web, native app, and hybrid applications. David also covers device-level testing, types of testing, available automation tools, and recommendations for automation effectiveness. Finally, based on his years of mobile testing experience, David provides some tips and tricks to approach mobile automation. Leave with a clear plan for automating your mobile applications.
Cultural Intelligence: A Key Skill for SuccessTechWell
Diversity is becoming the norm in everyday life. However, introducing global delivery models without a proper understanding of intercultural differences can lead to difficulty, frustration, and reduced productivity. Priyanka Sharma and Thena Barry say that in our diverse world, we need teams with people who can cross these boundaries, communicate effectively, and build the diverse networks necessary to avoid problems. We need to learn about cultural intelligence (CI) and cultural quotient (CQ). CI is the ability to relate and work effectively across cultures. CQ is the cognitive, motivational, and behavioral capacity to understand and respond to beliefs, values, attitudes, and behaviors of individuals and groups. Together, CI and CQ can help us build behavioral capacities that aid motivation, behavior, and productivity in teams as well as individuals. Priyanka and Thena show how to build a more culturally intelligent place with tools and techniques from Leading with Cultural Intelligence, as well as content from the Hofstede cultural model. In addition, they illustrate the model with real-life experiences and demonstrate how they adapted in similar circumstances.
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
Why would a century-old utility with no direct competitors take on the challenge of transforming its entire IT application organization to an agile methodology? In an increasingly interconnected world, the expectations of customers continue to evolve. From smart meters to smart phones, IoT is creating a crisis point for industries not accustomed to rapid change. Glen Morris explains that pizzas can be tracked by the minute and packages at every stop, and customers now expect this same customer service model should exist for all industries—including power. Glen examines how to create momentum and transform non-IT-focused industries to an agile model. If you are struggling with gaining traction in your pursuit of agile within your business, Glen gives you concrete, practical experiences to leverage in your pursuit. Finally, he communicates how to gain buy-in from business partners who have no idea or concern about agile or its methodologies. If your business partners look at you with amusement when you mention the need for a dedicated Product Owner, join Glen as he walks you through the approaches to overcoming agile skepticism.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxSitimaJohn
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Skybuffer SAM4U tool for SAP license adoptionTatiana Kojar
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Project Management Semester Long Project - Acuityjpupo2018
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
1. MQ
PM Tutorial
9/30/2013 1:00:00 PM
"How to Break Software:
Embedded Edition"
Presented by:
Jon Hagar
Consultant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Jon Hagar
Grand Software Testing
Jon Hagar is a systems-software engineer and tester consultant supporting software product
integrity and verification and validation, with a specialization in embedded and mobile software
systems. For more than thirty years Jon has worked in software engineering, particularly testing,
supporting projects including control system (avionics and auto), spacecraft, mobile-smart
devices, IT, and attack testing of smart phones.
3. 8/20/2013
How to Break Software:
Embedded Edition
Attacks to find common bugs
quickly in embedded software
systems
Jon D. Hagar
embedded@ecentral.com
Jon.d.hagar@gmail.com
http://www.breakingembeddedsoftware.com/
Jon Hagar Copy right 2013
1
Some Thoughts
What are our objectives?
◦ Definitions and introductions
◦ Understand Applicable Mobile and Embedded
Test Concepts
◦ Practice some testing
Jon Hagar Copy right 2013
2
Software Test Attacks To Break Mobile and Embedded Devices
1
4. 8/20/2013
This is a Workshop Tutorial
A bit of a talking head (with charts)
◦ Based on my book
Attendees should be prepared to:
◦
◦
◦
◦
Do some reading & thinking
Use the reference material
Talk & ask questions
Share (lessons learned and retrospectives)
LET’S PLAY TEST… and try out some new things…..
It might get LOUD
Jon Hagar Copy right 2013
3
Software Test Attacks To Break Mobile and Embedded Devices
Agenda
Definitions and introductions
Risk-based concepts
Exploratory approaches
Attacking scenario(s)
Attacking the hardware-software interface
Wrap up and references
Jon Hagar Copy right 2013
4
Software Test Attacks To Break Mobile and Embedded Devices
2
5. 8/20/2013
Definitions
Embedded Software Systems . . .
Interact with unique hardware/systems to solve specialized
problems in the “real world”
◦ IT software runs with largely “generic” hardware
◦ Users are barely aware the device uses or has software
Usually have significant hardware interface issues and concerns
◦ Initialization, noise, power up/down, timers, sensors, etc.
Often are resource constrained
◦ RAM, ROM, stack, power, speed, time, etc.
Typically has a restricted or no Human User/Computer Interface
(HCI) but is evolving rapidly
Often no way (or only a risky way) to update and/or change the
software
Involves risks, hazards, safety, and/or some specialized domain
knowledge and logic/algorithms usually controlling hardware
Jon Hagar Copy right 2013
5
Software Test Attacks To Break Mobile and Embedded Devices
Close Cousins: Mobile, Smart, &
Handheld
As the names imply, these are devices—small, held in the hand, often
connected to communication networks, including:
◦ Cell and smart phones – apps (not covered today)
◦ Tablets
◦ Medical devices
Typically have:
◦ Many of the problems of classic “embedded” systems
◦ The power of PCs/IT
◦ More user interface (UI) than classic embedded systems
◦ Fast updates
Are getting more powerful, memory and features (software, e.g., apps)
The “hot” area of computers/software
◦ Testing rules are “evolving”
Jon Hagar Copy right 2013
6
Software Test Attacks To Break Mobile and Embedded Devices
3
6. 8/20/2013
What do these look like?
Examples
– Avionics systems: planes, cars, rockets, military. . .
– Telecom: switch, routers, phones, cell devices
– Transportation: traffic control, railroad, trucking
– Industrial control: lighting, machines, HVAC, nuclear/power
– Medical: pacemaker, AEDs, defibrulators, dispensers, etc.
– Home and office systems: control, entertainment (TV box)
– And the list goes on
• Now starting to include PDA’s and other items that “blur”
the lines
Jon Hagar Copy right 2013
7
Software Test Attacks To Break Mobile and Embedded Devices
Fundamental Software Capabilities Defined
Dr. James Whittaker lists four capabilities:
1. Software accepts inputs from its environment
2. Software produces output and transmits it to its
environment
3. Software stores data internally in one or more
data structures
4. Software performs computations using input or
stored data
Embedded devices can be refined with
◦
◦
◦
Function in/with Time
Use/control of unique hardware, OR
Real world specialization of items 1 and 2
Jon Hagar Copy right 2013
8
Software Test Attacks To Break Mobile and Embedded Devices
4
7. 8/20/2013
Knowing the Bug (error) - Defined
•
Handheld/Embedded software has
similar defects to other software
• Requirements & Design
• Logic & Math
• Control Flow
• Data
• Initialization & Mode changes
• Interfaces
• Security
• Gaming
• etc. . .
But adds context defects/issues
• Software and hardware
development cycles done in
parallel, where aspects of the
hardware may be unknown to the
software development effort
• Hardware problems which are
often fixed with software late in
the project
• Small amounts of dense complex
functions often in the control
theory or safety/hazard domains
• (a big one) Very tight real-time
performance issues (often in millior micro-second ranges)
Jon Hagar Copy right 2013
9
Software Test Attacks To Break Mobile and Embedded Devices
“World” of Mobile-Smart/Embedded Software
Response-Outputs
Stimulus-Inputs
Expected
Unexpected
Wanted
Hardware
Unwanted
Software
Inputs and outputs involve hardware, software, and
humans
Time dependent
◦ NOTE: most software has “time” (performance) issues but here
things are often “hard real time”
◦ Embedded and real-time “time” may be a requirement
Jon Hagar Copy right 2013
10
Software Test Attacks To Break Mobile and Embedded Devices
5
8. 8/20/2013
Exercise: Why do we test?
Handheld Mobile/Embedded Software (or any
software)?
Jon Hagar Copy right 2013
11
Software Test Attacks To Break Mobile and Embedded Devices
YAM* Lifecycle Embedded (*yet another model)
Software - Many builds, iterations and increments
Test “circles” around schedule milestones
start
Lab drop
end
Build 1
start
Eng drop lab drop
end
Build 2
…………
…Prototype………
………Prototype n…..
Test Efforts
But what about the hardware lifecycle?
Jon Hagar Copy right 2013
12
Software Test Attacks To Break Mobile and Embedded Devices
6
9. 8/20/2013
Example High Level Embedded Lifecycle
System Creation
Hardware Build
Hardware Build
Hardware Build
Hw
Issue
Software Build
Software Build
Software Build
Test/V&V
Software Build
Software Build
Results: Software is “late”
Jon Hagar Copy right 2013
13
Software Test Attacks To Break Mobile and Embedded Devices
My Assumptions…
This is not a “general” class on systems,
software, and/or testing and
I assume the following knowledge:
◦
◦
◦
◦
◦
◦
◦
Test plans and planning
Requirements testing
Test labs and building labs
Standards you operate under (yes, there are many)
Tools you use
Testing experience (a software system)
Embedded design for testability is an accepted
practice
That you want something more . . .
Jon Hagar Copy right 2013
14
Software Test Attacks To Break Mobile and Embedded Devices
7
10. 8/20/2013
If What I Assume is False…
(when you get home)
Reference list is available to do some reading
Other full classes are available
You are reading books
You will ask questions
You are looking to have an epiphany
You are ready to learn
Keep in mind that I do not have all the answers
Jon Hagar Copy right 2013
15
Software Test Attacks To Break Mobile and Embedded Devices
Section 1: Testing
Preliminary
Jon Hagar Copy right 2013
16
8
11. 8/20/2013
Exercise: Test the Embedded Game
Break into teams
Load the 20Q app on smart phone (if you want)
Define a test
Define some rules: No destructive testing please
List of requirements
◦ This is a handheld game
◦ You think of something (say spinach) and it figures out what you are thinking
by proposing 20 questions to you
◦ Questions begin with animal, vegetable, mineral and go from there
◦ Game has non-standard input keys, display screen, and embedded software
◦ Game knows things and will figure out what you are thinking of
Now . . . build a test for this device
Jon Hagar Copy right 2013
17
Software Test Attacks To Break Mobile and Embedded Devices
What do you mean you do not have
enough?
What is wrong?
What do you need to do testing?
Is this not the world many testers live in?
We should start simple in testing, but maybe
this is not simple enough?
Jon Hagar Copy right 2013
18
Software Test Attacks To Break Mobile and Embedded Devices
9
12. 8/20/2013
So, Let’s Back up a Little
Let me give you some attack support concepts &
techniques (in case you don’t know these)
You can apply these if you are a staff tester or a
“crowd source” contractor
This is a simulation, but in the real world, often you
will just be given the software or a device to test --- You CAN test…….
Jon Hagar Copy right 2013
19
Software Test Attacks To Break Mobile and Embedded Devices
Risk and Exploratory-Attack Testing
You cannot test everything
Risk(s) based testing helps bound the test scope
problem
Testing is about providing information and understanding
Exploration gets you started with whatever you have (or
don’t have)
Jon Hagar Copy right 2013
20
Software Test Attacks To Break Mobile and Embedded Devices
10
13. 8/20/2013
Basic to Attacks: Risk-Based Testing
Address, mitigate, attack and retire product
risks
Do you remember what a risk is?
◦ Potential problem - consequence and effect
◦ Occurrence – likelihood or chance of happening
◦ Impact – what happens
Do this from the beginning (proposal) to the
end (retirement) of the product (Hw-Sw)
lifecycle
Risks should feed the Attacks (more on that
later)
Jon Hagar Copy right 2013
21
Software Test Attacks To Break Mobile and Embedded Devices
Sample Product Risks Testers Should
Consider
Safety
Security
Hazard
Business impacts
Control (loss of. . . )
Computation
Functional elements
Non-functional
Data
Regulation (s) and legal factors
Output noise
Environment and input factors
System factors – complexity, interfaces, human/non-human
Jon Hagar Copy right 2013
22
Software Test Attacks To Break Mobile and Embedded Devices
11
14. 8/20/2013
How to Use Risk Analysis in Testing
Goal oriented testing (where to focus)
Priority of attack and scenario
◦ Never enough time to test everything
◦ Can define the “un attacked” (risks)
Minimization of risks by focusing on the scary or
critical first
Provide information back to the team sooner
Jon Hagar Copy right 2013
23
Software Test Attacks To Break Mobile and Embedded Devices
Risk-Based Testing Process (simple)
Identify the product
Find product supporting information
Identify risks associated with the product
Risk priority (what you will test first?)
The resulting risks by priority define the
attacks
Jon Hagar Copy right 2013
24
Software Test Attacks To Break Mobile and Embedded Devices
12
15. 8/20/2013
Risk Analysis Throughout the Test Process
Many testers just think “requirements” in embedded,
but…
Always be “thinking” risks, since it can drive and control
your testing
Do this by team brainstorming (make lists)
Tests and analysis provide learning/data
points/information
◦ Errors in an area of code?
◦ Hardware that doesn’t work?
◦ Piece of code from a vendor is more complex?
◦ Operations the system will/can do?
Particularly off nominal and unusual (where bugs hide)
Jon Hagar Copy right 2013
25
Software Test Attacks To Break Mobile and Embedded Devices
Exercise: Redeaux
Back into teams
Conduct a risk exercise for your device
Risk Statement ( If x, then y happens)
Jon Hagar Copy right 2013
Priority
26
Software Test Attacks To Break Mobile and Embedded Devices
13
16. 8/20/2013
Risks Should Define
Exploration
For mobile-embedded, exploratory testing
can be important
Jon Hagar Copy right 2013
27
Exploratory – Attack Testing
What is it?
◦
◦
◦
◦
Scientific “methods”
Engineering understanding
May call it something else, but most of us do it
Attacks “target” specific bugs using test techniques
How and when to apply?
◦ As early in a lifecycle as possible (with prototypes, models,
etc.)
◦ When you want to “learn” and test at the same time
◦ When being a little “informal” is OK
◦ All the time?
Jon Hagar Copy right 2013
28
Software Test Attacks To Break Mobile and Embedded Devices
14
17. 8/20/2013
Exploratory–Attack Testing Definition
Bach/Kaner:
“Exploratory testing is simultaneous learning,
test design and test execution.”
Exploratory testing has rules and concepts
Underlying it is a “model” of human understanding of
software and knowing how that fails
NOT AD HOC: Ad hoc has all too often been associated
with sloppiness, carelessness, no documentation, nonrepeatable, and so forth—but may have its place at times
Jon Hagar Copy right 2013
29
Software Test Attacks To Break Mobile and Embedded Devices
In Embedded
Exploratory testing is situational - Use it when…
Need rapid feedback
Learning
Upfront rapid learning
Attacking
There are risks
Need independent assessment
Targeting a defect
Prototyping
Need info
Testing beyond the requirements
Jon Hagar Copy right 2013
30
Software Test Attacks To Break Mobile and Embedded Devices
15
18. 8/20/2013
General Concept of Exploratory
Many authors define it as:
Time/Schedule (limited)
The Tester (your team)
A Testing Mission (also called “Charter”)
Results
Usually in the form of opened Defects
Sometimes an annotated Mission statement and
opened Defects list
Maybe a “report”
Retrospective (more on that in a minute)
Jon Hagar Copy right 2013
31
Software Test Attacks To Break Mobile and Embedded Devices
Exploratory Critical Components
Test Design
Critical Thinking
Diverse Ideas
Rich Resources
Careful Observation
Jon Hagar Copy right 2013
32
Software Test Attacks To Break Mobile and Embedded Devices
16
19. 8/20/2013
Pattern for This Class (one of many)
Have an outline (top level plan and/or risk list)
Create a flip chart, notecard, state model, or some
representation of each test task
◦ No “heavy” weight documentation of the “test case”
◦ See Exploratory Charter (test objective)
Have a Target concept or charter (Risk, Attack, Bug, Learning,
etc.)
Have a schedule/time box (hours — not more than 1-2 days)
Do the test
◦ Design test
◦ Execute test
◦ Learn about the product: change the risk list, modify/add
tests, and so on
Repeat the process as needed
Jon Hagar Copy right 2013
33
Software Test Attacks To Break Mobile and Embedded Devices
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
◦ Risk (s):
Success Criteria:
1.
◦ Attack
2.
◦
3.
Other (requirements, …..)
Support items needed:
Role (User you play during the test):
Actions:
◦ 1.
◦ 2.
◦ 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Document all of these.
Jon Hagar Copy right 2013
34
Software Test Attacks To Break Mobile and Embedded Devices
17
20. 8/20/2013
Exercise: So Let’s Go Back to
the Embedded Game Device to Do Testing
Test of the Game App
Use the risks (chart 25) for the device to define test
objectives
Apply Exploratory-Attack
◦ Do a Charter
Learn – do one cycle of exploration
Jon Hagar Copy right 2013
35
Software Test Attacks To Break Mobile and Embedded Devices
Group Flip Chart
Feedback - Retrospective
What did you accomplish?
◦ Did you find any bugs? If so, how many?
What did you think of?
What would you do differently?
Jon Hagar Copy right 2013
36
Software Test Attacks To Break Mobile and Embedded Devices
18
21. 8/20/2013
Section 2:
So we have Risk Analysis &
have done a first Exploratory Test
Basic and addressed in many books
What’s next?
Lets get to different levels of testing, embedded
devices, and more fun
Jon Hagar Copy right 2013
37
What is an Attack
Attacking your software–In part, the process of attempting
to demonstrate that a system (hardware, software, and
operations) does not meet requirements, functional and
non-functional objectives
◦ Embedded/handheld software testing must include “the system”
(hardware, software, operations, users, etc.)
Attacks go after common modes of failure and bugs to
demonstrate that “does not meet” exists
We go after our enemy with many approaches
Jon Hagar Copy right 2013
◦
◦
◦
◦
◦
Tools
Levels
Attacks
Techniques
Etc.
38
Software Test Attacks To Break Mobile and Embedded Devices
19
22. 8/20/2013
An Attack Is…
Based on a common mode of failure seen over and
over
◦ Maybe seen as a negative, when it really is a positive
◦ Goes after the “bugs” that may be in the software
◦ Based on or using classic test techniques and test concepts
Lee Copeland’s book on test design
Many other good books
A Pattern (more than a process) which must be
modified for the context at hand to do the testing
Testers learn these in a domain after years and form a
mental model (most good testers attack)
Jon Hagar Copy right 2013
39
Software Test Attacks To Break Mobile and Embedded Devices
Kinds of Attacks
Whittaker offers a good starting point for software
attacks in general that can be applied to embedded:
◦ User Interface Attacks
◦ Data and Computation
◦ File System Interface
◦ Software/OS Interface
Whittaker’s “How to Break Software” lists 23 attacks
“Software Test Attacks to Break Mobile and
Embedded Devices” (my book) adds 32 attacks and 8
sub attacks
Jon Hagar Copy right 2013
40
Software Test Attacks To Break Mobile and Embedded Devices
20
23. 8/20/2013
Embedded Attack Classification
Developer Attacks (unit/code testing)
Control System Attacks
Hardware-Software Attacks
Mobile and Embedded Software Domain Attacks
Time Attacks (Performance)
Human User Interface Attacks
Smart and/or Mobile Phone Functional App Attacks
Mobile/Embedded Security Attacks
Generic Attacks
◦ Functional, mind mapping, and combinatorial tests
Jon Hagar Copy right 2013
41
Software Test Attacks To Break Mobile and Embedded Devices
The Software You Test
Do you know how it fails?
Do you test for success or failure or both?
Will this workshop give you all the answers and all possible
attacks?
◦ No, but you can start asking questions and thinking
Jon Hagar Copy right 2013
42
Software Test Attacks To Break Mobile and Embedded Devices
21
24. 8/20/2013
Introducing the Robots
Requirements – in the class hand out for each robot grouping
Rules (this exercise takes some thinking and reading)
◦ NO destructive testing (Please BE CAREFUL with the robots)
◦ There are bugs to be found (record and report them)
◦ Each group defines an attack and gets “time” on the devices (but time in our
environment is limited—just as it is in the real world)
Environment
◦ This room, but we have some “test tools”
◦ This is software testing, but within the hardware it is embedded in
This will be a simple testing process, but use the attack concepts and report
experiences back in the debrief
1. Define risks based on hardware, software, requirements, and bugs (risk list)
2. Conduct each attack session using a charter
3. Define a test attack using provided attack pattern (handout)
4. Will do a debrief after each test session
5. Group will rotate different robot configurations
Jon Hagar Copy right 2013
43
Software Test Attacks To Break Mobile and Embedded Devices
Additional Considerations
We will try to run at least 2 different attacks
Use the concepts we used on games
Ask questions (of each other & me)
No destructive testing and I am the “tester” (you
must tell me what to do)
Use the tools at hand
But first we need to think about embedded users
– next exercise
Jon Hagar Copy right 2013
44
Software Test Attacks To Break Mobile and Embedded Devices
22
25. 8/20/2013
More Test Time
We are going to do another attack on a different
embedded device
Each group will be tasked with one of the two attacks
Each group should complete a charter and see the “test
master” for access to the device
You’ll have a robot with software loaded
Practice identifying: Risks, Users, Exploration, and Attacks
Follow the “suggested patterns” of the attacks
We’ll go until no time left
Jon Hagar Copy right 2013
45
Software Test Attacks To Break Mobile and Embedded Devices
Exercise: Understanding Users for
Embedded Attacks
Let’s list some users of the games and
robots because . . .
◦ Users play into risks, attacks, bugs, what to look for
◦ You should be able to do the same for the robots (or
any software you test)
Jon Hagar Copy right 2013
46
Software Test Attacks To Break Mobile and Embedded Devices
23
26. 8/20/2013
Exercise Answer: List Users
Jon Hagar Copy right 2013
47
Software Test Attacks To Break Mobile and Embedded Devices
Now you have a few basics:
•
•
•
•
Jon Hagar Copy right 2013
Risk thinking
Exploration
Software’s users
Attack patterns are provided next
48
24
27. 8/20/2013
Attack Group 15
Scenarios and actions over time
Jon Hagar Copy right 2013
49
Software Test Attacks To Break Mobile and Embedded Devices
Attack Stories, Tours, and Scenarios
Call them what you will
There are subtle differences depending on whose
material you have read
They are how the system gets used end-to-end
They combine use, users, information, techniques,
tools, and (maybe) attacks
Jon Hagar Copy right 2013
50
Software Test Attacks To Break Mobile and Embedded Devices
25
28. 8/20/2013
Apply This Attack When…
Time interacts with the software, events, inputs,
and outputs
Checklist of things to look for and consider
(possible bugs)
◦ Order problems
◦ Too Long
◦ Too Fast
◦ Not at Right Time mark or point
◦ Late
◦ Late or early
◦ Early
◦ Deadlocked caused by a race condition(hard to find)
◦ Extra input or output events
◦ Missing events
◦ Wrong input/output within events
Jon Hagar Copy right 2013
51
Software Test Attacks To Break Mobile and Embedded Devices
Attack Factors
What - Look for things not in the right order
Who – Test team
Where – Lab and/or field testing where hardware
and software interact
◦ Tools may be important here
Jon Hagar Copy right 2013
52
Software Test Attacks To Break Mobile and Embedded Devices
26
29. 8/20/2013
How (a generic pattern)?
Understand what the system does or is supposed to do
◦ a sequence of events or functions
◦ Look in: concepts of operations, user guides, use cases, models, & any other
information that will detail functions and usage over time
◦ From these, organize a sequence or set of sequences
First attack case: Focus on a typical situation based on requirements and/or use
cases
Second attack case: Consider the off–normal, non–failure modes
◦ Look for the failure modes and effects—does the software recover well?
◦ Review and understand system errors and failure history from the field
Build up histories of attacks based on outputs and log files
◦ Warning: log files can contain large amounts of detailed data and this can
also adversely affect the performance (especially timing) of the software
Conduct risk analysis as the effort progresses
Final Attack cases: Build Extreme cases such as “Soap Opera” Tests
Warning: Watch becoming “script” bound
Jon Hagar Copy right 2013
53
Software Test Attacks To Break Mobile and Embedded Devices
Exercise – Tell Me Stories
Work in groups
Key points
◦ Define a “story/scenario” with a name and outline
◦ Use the check list on chart 51 (note what is used)
◦ Follow pattern of chart 52 (note steps on charter)
Can you build a Tour (combination of story patterns)
Products
◦
◦
◦
◦
Risk list
User list
Test charter
Note bugs (if any)
Complete the feedback retrospective
Do several attacks (note what each one is)
Expand to “extreme cases”
Jon Hagar Copy right 2013
54
Software Test Attacks To Break Mobile and Embedded Devices
27
30. 8/20/2013
Attack 7
Digitals v Analog Integration
Jon Hagar Copy right 2013
55
Software Test Attacks To Break Mobile and Embedded Devices
Attack Hw-Sw Interface Group
Here we are attacking the hardware-tosoftware interface
Many bugs
- Developers
hide in the interfaces
often miss these
Jon Hagar Copy right 2013
56
Software Test Attacks To Break Mobile and Embedded Devices
28
31. 8/20/2013
Attack: Analog-Digital Hw-Sw Interfaces
When - The software is “controlling” the unique
hardware
What – Look at the interface, hardware (as a user), and
what the software is controlling
Who – Test team (independent)
Where – Lab where the hardware and software are both
present
Bugs to look for (next page)
Jon Hagar Copy right 2013
57
Software Test Attacks To Break Mobile and Embedded Devices
Taxonomy: A2D and D2A Bug
Possibilities
Type
A2D
A2D
A2D
D2A
D2A
D2A
Situation
Impact
A2D representation information is lost Software
because measurement is not precise computation is
based on incorrect
data
A2D information is contaminated with Software
noise
computation use
noise when it
should not
A2D information is calculated
Computation has
correctly
unknown error
D2A conversion losses “least
significant bits” (LSB) in conversions,
but bits are, in fact, important
because computer word sizes are too
small
D2A information does not account for
noise of the real world
D2A information is calculated
correctly because of internal factors
Output to analog
device is wrong
Software
computation does
not include a factor
for noise
Computation has
unknown error
Notes
Number of bits used to store the analog
converted data is not large enough or
sampling rate to get bits is not correct.
The noise term may not be known,
accounted for, or misrepresented.
Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point
Number of bits stored from the digital
world to the analog world do not have
enough precision, so analog data is
incorrect.
The analog values are not correct given the
noise of the real world (output data may be
lost in the noise).
Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point
Jon Hagar Copy right 2013
58
Software Test Attacks To Break Mobile and Embedded Devices
29
32. 8/20/2013
How?
Up front data gathering and analysis are important beginnings – what do we
know (or can ask about)
Identify input devices
Identify output devices
Define the input disturbances (unexpected system inputs)
Define possible output disturbances (unexpected system outputs)
Determine what is or is not possible in the test environment
Conduct a risk analysis (see likely bugs table)
Identify the users of the device and software
- Testers should be aware that embedded systems have resource
constraints in memory, CPU usage, and time
Use the above information to define an exploratory chart attack
Go run that attack
Learn
Design
Repeat (until time runs out)
Think!
Jon Hagar Copy right 2013
59
Software Test Attacks To Break Mobile and Embedded Devices
Questions to Ask with this Attack
If the hardware is a prototype (not like what will be in the field), will
that impact testing or test results?
If a simulation is used, what bugs might be missed because actual
hardware or software is not used?
If the test inputs and environment are not representative of the real
world both in terms of expected and unexpected values, what risks
will be acceptable?
If the hardware is not understood, will testing be weak?
If the major sources of “noise” is not defined, will the system be
susceptible to impacts from unexpected inputs or outputs?
All of these questions will involve test tradeoffs, acceptable risk, and
compromise
◦ 40% of this kind of attack should be “normal” situations
◦ Start with normal and move to off-normal and stress cases
Jon Hagar Copy right 2013
60
Software Test Attacks To Break Mobile and Embedded Devices
30
33. 8/20/2013
Exercise – A2D/D2A
Work in groups
Key points
◦ Think how to look for bugs of chart 58
◦ Use the questions on chart 60 (note what is used)
◦ Follow pattern of chart 59 (note steps on charter)
Can you expand with stories/tours?
Products
◦
◦
◦
◦
Risk list
User list
Test charter
Note bugs (if any)
Complete the feedback retrospective
Do several attacks (note what each one is)
Expand to “extreme cases”
Jon Hagar Copy right 2013
61
Software Test Attacks To Break Mobile and Embedded Devices
Feedback – Retrospective Session
What did you accomplish?
◦ Bugs?
◦ Tests?
What things did you think of?
◦ Wish I had a _____???? I need more time???
What favors or opposes an attack?
What would you do differently next cycle?
Jon Hagar Copy right 2013
62
Software Test Attacks To Break Mobile and Embedded Devices
31
34. 8/20/2013
Final Thoughts . . .
Jon Hagar Copy right 2013
63
Wrap Up
This tutorial covers some basic introduction
(key attacks) and sampling
◦ There are many more
Understanding your local context and error
patterns is important (one size does NOT fit all)
Attacks are patterns…you still must THINK!
These attacks target Embedded and Mobile
Jon Hagar Copy right 2013
64
Software Test Attacks To Break Mobile and Embedded Devices
32
35. 8/20/2013
More Attacks (from my book and others)
Attack 1: Static Code Analysis
Attack 18: Bugs in Timing Interrupts and Priority Inversion
Attack 2: Finding White–Box Data Computation Bugs
Attack 19: Finding Time Related Bugs
Attack 3: White–Box Structural Logic Flow Coverage
Attack 20: Time Related Scenarios, Stories and Tours
Attack 4: Finding Hardware–System Unhandled Uses in Software
Attack 21: Performance Testing Introduction
Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs
Attack 22: Finding Supporting (User) Documentation Problems
Sub–Attack 22.1: Confirming Install–ability
Attack 6: Long Duration Control Attack Runs
Attack 23: Finding Missing or Wrong Alarms
Attack 7: Breaking Software Logic and/or Control Laws
Attack 24: Finding Bugs in Help Files
Attack 8: Forcing the Unusual Bug Cases
Attack 25: Finding Bugs in Apps
Attack 9 Breaking Software with Hardware and System
Operations
Attack 26: Testing Mobile and Embedded Games
Attack 27: Attacking App–Cloud Dependencies
9.1 Sub–Attack: Breaking Battery Power
Attack 28 Penetration Attack Test
Attack 10: Finding Bugs in Hardware–Software Communications
Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack
Attack 11: Breaking Software Error Recovery
Attack 28.2 Sub–Attack Fuzz Test
Attack 29: Information Theft—Stealing Device Data
Attack 12: Interface and Integration Testing
Attack 29.1 Sub Attack –Identity Social Engineering
12.1 Sub–Attack: Configuration Integration Evaluation
Attack 30: Spoofing Attacks
Attack 13: Finding Problems in Software–System Fault Tolerance
Attack 30.1 Location and/or User Profile Spoof Sub–Attack
Attack 14: Breaking Digital Software Communications
Attack 30.2 GPS Spoof Sub–Attack
Attack 15: Finding Bugs in the Data
Attack 31: Attacking Viruses on the Run in Factories or PLCs
Attack 16: Bugs in System–Software Computation
Attack 32: Using Combinatorial Tests
Attack 17: Using Simulation and Stimulation to Drive Software
Attacks
Attack 33: Attacking Functional Bugs
Jon Hagar Copy right 2013
65
Software Test Attacks To Break Mobile and Embedded Devices
Summary: Thank You (ideas used from)
James Whittaker (attacks)
Elisabeth Hendrickson (simulations)
Lee Copeland (techniques)
Brian Merrick (testing)
James Bach (exploratory & tours)
Cem Kaner (test thinking)
Many teachers
Generations past and future
Books, references, etc.
Jon Hagar Copy right 2013
66
Software Test Attacks To Break Mobile and Embedded Devices
33
36. 8/20/2013
Book List (my favorites)
“Software Test Attacks to Break Mobile and Embedded Devices”
“How to Break Software” James Whittaker, 2003
◦ And his other “How To Break…” books
“Testing Embedded Software” Broeckman and Notenboom, 2003
“A Practitioner’s Guide to Software Test Design” Copeland, 2004
“A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993
“Computer Related Risks”, Neumann, 1995
“Safeware: System Safety and Computers”, Leveson, 1995
Honorable mentions:
◦ “Embedded System and Software Validation” Roychoudhury, 2009
◦ “Systems Testing with an Attitude” Petschenik 2005
◦ “Software System Testing and Quality Assurance” Beizer, 1987
◦ “Testing Computer Software” Kaner et. al., 1988
◦ “Systematic Software Testing” Craig & Jaskiel, 2001
◦ “Managing the Testing Process” Black, 2002
– Jon Duncan Hagar, due out late 2013
(http://www.crcpress.com/product/isbn/9781466575301)
Jon Hagar Copy right 2013
67
Software Test Attacks To Break Mobile and Embedded Devices
More Resources
•
www.stickyminds.com – Collection of test info
My Web site:
www.breakingembeddedsoftware.com
•
Association of Software Testing
•
– BBST Classes http://www.testingeducation.org/BBST/
•
Your favorite search engine
Jon Hagar Copy right 2013
68
Software Test Attacks To Break Mobile and Embedded Devices
34
37. 8/20/2013
Definitions (for this class)
Taxonomy – the practice and science of classification.
Test – the act of conducting experiments on something to determine the
quality and provide information
Test case – one set of inputs, environmental set up, and results (expected
and unexpected)
Attack – to set up, forcefully and attempt to “damage” the system or
software, using tools, methods, and techniques
Bug (error) – results that depart from the expected (from requirements,
design, standards, user, etc.)
Lifecycle – from beginning-to-end, the steps, stages, and activities to create
(birth-to-death)
Procedure – a particular way of accomplishing tests, usually written (one or
more test cases)
Tour – a journey to find information (tests) with a focus/direction (story)
Scenario – a sequence of events with a test plot or story
Script – see procedure, but normally uses automation
Users – someone/something that interacts with the system/software (can be
human or machine, or?)
Quality – value to someone and that they will pay for
Jon Hagar Copy right 2013
69
Software Test Attacks To Break Mobile and Embedded Devices
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
– Other (requirements, …..)
•
3.
•
Support items needed:
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
•
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copy right 2013
70
Software Test Attacks To Break Mobile and Embedded Devices
35
38. 8/20/2013
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
– Other (requirements, …..)
•
3.
•
Support items needed:
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
•
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copy right 2013
71
Software Test Attacks To Break Mobile and Embedded Devices
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
– Other (requirements, …..)
•
3.
•
Support items needed:
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
•
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copy right 2013
72
Software Test Attacks To Break Mobile and Embedded Devices
36
39. 8/20/2013
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
– Other (requirements, …..)
•
3.
•
Support items needed:
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
•
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copy right 2013
73
Software Test Attacks To Break Mobile and Embedded Devices
Exploratory Test Card (Charter)
Name of Test:
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
– Other (requirements, …..)
•
3.
•
Support items needed:
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
•
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copy right 2013
74
Software Test Attacks To Break Mobile and Embedded Devices
37