How to Break Software: Embedded Edition
Upcoming SlideShare
Loading in...5
×
 

How to Break Software: Embedded Edition

on

  • 350 views

In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain ...

In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, he shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, attendees get to practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.

Statistics

Views

Total Views
350
Views on SlideShare
348
Embed Views
2

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 2

http://www.stickyminds.com 1
http://www.agileconnection.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

How to Break Software: Embedded Edition How to Break Software: Embedded Edition Document Transcript

  • MQ PM Tutorial 9/30/2013 1:00:00 PM "How to Break Software: Embedded Edition" Presented by: Jon Hagar Consultant Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • Jon Hagar Grand Software Testing Jon Hagar is a systems-software engineer and tester consultant supporting software product integrity and verification and validation, with a specialization in embedded and mobile software systems. For more than thirty years Jon has worked in software engineering, particularly testing, supporting projects including control system (avionics and auto), spacecraft, mobile-smart devices, IT, and attack testing of smart phones.
  • 8/20/2013 How to Break Software: Embedded Edition Attacks to find common bugs quickly in embedded software systems Jon D. Hagar embedded@ecentral.com Jon.d.hagar@gmail.com http://www.breakingembeddedsoftware.com/ Jon Hagar Copy right 2013 1 Some Thoughts  What are our objectives? ◦ Definitions and introductions ◦ Understand Applicable Mobile and Embedded Test Concepts ◦ Practice some testing Jon Hagar Copy right 2013 2 Software Test Attacks To Break Mobile and Embedded Devices 1
  • 8/20/2013 This is a Workshop Tutorial  A bit of a talking head (with charts) ◦ Based on my book  Attendees should be prepared to: ◦ ◦ ◦ ◦  Do some reading & thinking Use the reference material Talk & ask questions Share (lessons learned and retrospectives) LET’S PLAY TEST… and try out some new things…..  It might get LOUD Jon Hagar Copy right 2013 3 Software Test Attacks To Break Mobile and Embedded Devices Agenda  Definitions and introductions  Risk-based concepts  Exploratory approaches  Attacking scenario(s)  Attacking the hardware-software interface  Wrap up and references Jon Hagar Copy right 2013 4 Software Test Attacks To Break Mobile and Embedded Devices 2
  • 8/20/2013 Definitions Embedded Software Systems . . .       Interact with unique hardware/systems to solve specialized problems in the “real world” ◦ IT software runs with largely “generic” hardware ◦ Users are barely aware the device uses or has software Usually have significant hardware interface issues and concerns ◦ Initialization, noise, power up/down, timers, sensors, etc. Often are resource constrained ◦ RAM, ROM, stack, power, speed, time, etc. Typically has a restricted or no Human User/Computer Interface (HCI) but is evolving rapidly Often no way (or only a risky way) to update and/or change the software Involves risks, hazards, safety, and/or some specialized domain knowledge and logic/algorithms usually controlling hardware Jon Hagar Copy right 2013 5 Software Test Attacks To Break Mobile and Embedded Devices Close Cousins: Mobile, Smart, & Handheld As the names imply, these are devices—small, held in the hand, often connected to communication networks, including: ◦ Cell and smart phones – apps (not covered today) ◦ Tablets ◦ Medical devices  Typically have: ◦ Many of the problems of classic “embedded” systems ◦ The power of PCs/IT ◦ More user interface (UI) than classic embedded systems ◦ Fast updates  Are getting more powerful, memory and features (software, e.g., apps)  The “hot” area of computers/software ◦ Testing rules are “evolving”  Jon Hagar Copy right 2013 6 Software Test Attacks To Break Mobile and Embedded Devices 3
  • 8/20/2013 What do these look like? Examples – Avionics systems: planes, cars, rockets, military. . . – Telecom: switch, routers, phones, cell devices – Transportation: traffic control, railroad, trucking – Industrial control: lighting, machines, HVAC, nuclear/power – Medical: pacemaker, AEDs, defibrulators, dispensers, etc. – Home and office systems: control, entertainment (TV box) – And the list goes on • Now starting to include PDA’s and other items that “blur” the lines Jon Hagar Copy right 2013 7 Software Test Attacks To Break Mobile and Embedded Devices Fundamental Software Capabilities Defined Dr. James Whittaker lists four capabilities: 1. Software accepts inputs from its environment 2. Software produces output and transmits it to its environment 3. Software stores data internally in one or more data structures 4. Software performs computations using input or stored data Embedded devices can be refined with ◦ ◦ ◦ Function in/with Time Use/control of unique hardware, OR Real world specialization of items 1 and 2 Jon Hagar Copy right 2013 8 Software Test Attacks To Break Mobile and Embedded Devices 4
  • 8/20/2013 Knowing the Bug (error) - Defined • Handheld/Embedded software has similar defects to other software • Requirements & Design • Logic & Math • Control Flow • Data • Initialization & Mode changes • Interfaces • Security • Gaming • etc. . . But adds context defects/issues • Software and hardware development cycles done in parallel, where aspects of the hardware may be unknown to the software development effort • Hardware problems which are often fixed with software late in the project • Small amounts of dense complex functions often in the control theory or safety/hazard domains • (a big one) Very tight real-time performance issues (often in millior micro-second ranges) Jon Hagar Copy right 2013 9 Software Test Attacks To Break Mobile and Embedded Devices “World” of Mobile-Smart/Embedded Software Response-Outputs Stimulus-Inputs Expected Unexpected   Wanted Hardware Unwanted Software Inputs and outputs involve hardware, software, and humans Time dependent ◦ NOTE: most software has “time” (performance) issues but here things are often “hard real time” ◦ Embedded and real-time “time” may be a requirement Jon Hagar Copy right 2013 10 Software Test Attacks To Break Mobile and Embedded Devices 5
  • 8/20/2013 Exercise: Why do we test?  Handheld Mobile/Embedded Software (or any software)? Jon Hagar Copy right 2013 11 Software Test Attacks To Break Mobile and Embedded Devices YAM* Lifecycle Embedded (*yet another model)  Software - Many builds, iterations and increments  Test “circles” around schedule milestones start Lab drop end Build 1 start Eng drop lab drop end Build 2 ………… …Prototype……… ………Prototype n….. Test Efforts But what about the hardware lifecycle? Jon Hagar Copy right 2013 12 Software Test Attacks To Break Mobile and Embedded Devices 6
  • 8/20/2013 Example High Level Embedded Lifecycle System Creation Hardware Build Hardware Build Hardware Build Hw Issue Software Build Software Build Software Build Test/V&V Software Build Software Build Results: Software is “late” Jon Hagar Copy right 2013 13 Software Test Attacks To Break Mobile and Embedded Devices My Assumptions…    This is not a “general” class on systems, software, and/or testing and I assume the following knowledge: ◦ ◦ ◦ ◦ ◦ ◦ ◦ Test plans and planning Requirements testing Test labs and building labs Standards you operate under (yes, there are many) Tools you use Testing experience (a software system) Embedded design for testability is an accepted practice That you want something more . . . Jon Hagar Copy right 2013 14 Software Test Attacks To Break Mobile and Embedded Devices 7
  • 8/20/2013 If What I Assume is False… (when you get home)       Reference list is available to do some reading Other full classes are available You are reading books You will ask questions You are looking to have an epiphany You are ready to learn Keep in mind that I do not have all the answers Jon Hagar Copy right 2013 15 Software Test Attacks To Break Mobile and Embedded Devices Section 1: Testing Preliminary Jon Hagar Copy right 2013 16 8
  • 8/20/2013 Exercise: Test the Embedded Game      Break into teams Load the 20Q app on smart phone (if you want) Define a test Define some rules: No destructive testing please List of requirements ◦ This is a handheld game ◦ You think of something (say spinach) and it figures out what you are thinking by proposing 20 questions to you ◦ Questions begin with animal, vegetable, mineral and go from there ◦ Game has non-standard input keys, display screen, and embedded software ◦ Game knows things and will figure out what you are thinking of  Now . . . build a test for this device Jon Hagar Copy right 2013 17 Software Test Attacks To Break Mobile and Embedded Devices What do you mean you do not have enough?  What is wrong?  What do you need to do testing?  Is this not the world many testers live in?  We should start simple in testing, but maybe this is not simple enough? Jon Hagar Copy right 2013 18 Software Test Attacks To Break Mobile and Embedded Devices 9
  • 8/20/2013 So, Let’s Back up a Little  Let me give you some attack support concepts & techniques (in case you don’t know these)  You can apply these if you are a staff tester or a “crowd source” contractor  This is a simulation, but in the real world, often you will just be given the software or a device to test --- You CAN test……. Jon Hagar Copy right 2013 19 Software Test Attacks To Break Mobile and Embedded Devices Risk and Exploratory-Attack Testing  You cannot test everything  Risk(s) based testing helps bound the test scope problem  Testing is about providing information and understanding  Exploration gets you started with whatever you have (or don’t have) Jon Hagar Copy right 2013 20 Software Test Attacks To Break Mobile and Embedded Devices 10
  • 8/20/2013 Basic to Attacks: Risk-Based Testing  Address, mitigate, attack and retire product risks  Do you remember what a risk is? ◦ Potential problem - consequence and effect ◦ Occurrence – likelihood or chance of happening ◦ Impact – what happens  Do this from the beginning (proposal) to the end (retirement) of the product (Hw-Sw) lifecycle  Risks should feed the Attacks (more on that later) Jon Hagar Copy right 2013 21 Software Test Attacks To Break Mobile and Embedded Devices Sample Product Risks Testers Should Consider  Safety  Security  Hazard  Business impacts  Control (loss of. . . )  Computation  Functional elements  Non-functional  Data  Regulation (s) and legal factors  Output noise  Environment and input factors  System factors – complexity, interfaces, human/non-human Jon Hagar Copy right 2013 22 Software Test Attacks To Break Mobile and Embedded Devices 11
  • 8/20/2013 How to Use Risk Analysis in Testing  Goal oriented testing (where to focus)  Priority of attack and scenario ◦ Never enough time to test everything ◦ Can define the “un attacked” (risks)  Minimization of risks by focusing on the scary or critical first  Provide information back to the team sooner Jon Hagar Copy right 2013 23 Software Test Attacks To Break Mobile and Embedded Devices Risk-Based Testing Process (simple)  Identify the product  Find product supporting information  Identify risks associated with the product  Risk priority (what you will test first?)  The resulting risks by priority define the attacks Jon Hagar Copy right 2013 24 Software Test Attacks To Break Mobile and Embedded Devices 12
  • 8/20/2013 Risk Analysis Throughout the Test Process     Many testers just think “requirements” in embedded, but… Always be “thinking” risks, since it can drive and control your testing Do this by team brainstorming (make lists) Tests and analysis provide learning/data points/information ◦ Errors in an area of code? ◦ Hardware that doesn’t work? ◦ Piece of code from a vendor is more complex? ◦ Operations the system will/can do?  Particularly off nominal and unusual (where bugs hide) Jon Hagar Copy right 2013 25 Software Test Attacks To Break Mobile and Embedded Devices Exercise: Redeaux Back into teams  Conduct a risk exercise for your device  Risk Statement ( If x, then y happens) Jon Hagar Copy right 2013 Priority 26 Software Test Attacks To Break Mobile and Embedded Devices 13
  • 8/20/2013 Risks Should Define Exploration For mobile-embedded, exploratory testing can be important Jon Hagar Copy right 2013 27 Exploratory – Attack Testing  What is it? ◦ ◦ ◦ ◦  Scientific “methods” Engineering understanding May call it something else, but most of us do it Attacks “target” specific bugs using test techniques How and when to apply? ◦ As early in a lifecycle as possible (with prototypes, models, etc.) ◦ When you want to “learn” and test at the same time ◦ When being a little “informal” is OK ◦ All the time? Jon Hagar Copy right 2013 28 Software Test Attacks To Break Mobile and Embedded Devices 14
  • 8/20/2013 Exploratory–Attack Testing Definition Bach/Kaner: “Exploratory testing is simultaneous learning, test design and test execution.”  Exploratory testing has rules and concepts  Underlying it is a “model” of human understanding of software and knowing how that fails  NOT AD HOC: Ad hoc has all too often been associated with sloppiness, carelessness, no documentation, nonrepeatable, and so forth—but may have its place at times Jon Hagar Copy right 2013 29 Software Test Attacks To Break Mobile and Embedded Devices In Embedded  Exploratory testing is situational - Use it when…           Need rapid feedback Learning Upfront rapid learning Attacking There are risks Need independent assessment Targeting a defect Prototyping Need info Testing beyond the requirements Jon Hagar Copy right 2013 30 Software Test Attacks To Break Mobile and Embedded Devices 15
  • 8/20/2013 General Concept of Exploratory Many authors define it as:  Time/Schedule (limited)  The Tester (your team)  A Testing Mission (also called “Charter”)  Results  Usually in the form of opened Defects  Sometimes an annotated Mission statement and opened Defects list  Maybe a “report”  Retrospective (more on that in a minute) Jon Hagar Copy right 2013 31 Software Test Attacks To Break Mobile and Embedded Devices Exploratory Critical Components  Test Design  Critical Thinking  Diverse Ideas  Rich Resources  Careful Observation Jon Hagar Copy right 2013 32 Software Test Attacks To Break Mobile and Embedded Devices 16
  • 8/20/2013 Pattern for This Class (one of many)       Have an outline (top level plan and/or risk list) Create a flip chart, notecard, state model, or some representation of each test task ◦ No “heavy” weight documentation of the “test case” ◦ See Exploratory Charter (test objective) Have a Target concept or charter (Risk, Attack, Bug, Learning, etc.) Have a schedule/time box (hours — not more than 1-2 days) Do the test ◦ Design test ◦ Execute test ◦ Learn about the product: change the risk list, modify/add tests, and so on Repeat the process as needed Jon Hagar Copy right 2013 33 Software Test Attacks To Break Mobile and Embedded Devices Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: ◦ Risk (s): Success Criteria: 1. ◦ Attack 2. ◦  3. Other (requirements, …..)  Support items needed:  Role (User you play during the test): Actions: ◦ 1. ◦ 2. ◦ 3.  Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Document all of these. Jon Hagar Copy right 2013 34 Software Test Attacks To Break Mobile and Embedded Devices 17
  • 8/20/2013 Exercise: So Let’s Go Back to the Embedded Game Device to Do Testing  Test of the Game App  Use the risks (chart 25) for the device to define test objectives  Apply Exploratory-Attack ◦ Do a Charter  Learn – do one cycle of exploration Jon Hagar Copy right 2013 35 Software Test Attacks To Break Mobile and Embedded Devices Group Flip Chart Feedback - Retrospective  What did you accomplish? ◦ Did you find any bugs? If so, how many?  What did you think of?  What would you do differently? Jon Hagar Copy right 2013 36 Software Test Attacks To Break Mobile and Embedded Devices 18
  • 8/20/2013 Section 2: So we have Risk Analysis & have done a first Exploratory Test Basic and addressed in many books What’s next? Lets get to different levels of testing, embedded devices, and more fun Jon Hagar Copy right 2013 37 What is an Attack  Attacking your software–In part, the process of attempting to demonstrate that a system (hardware, software, and operations) does not meet requirements, functional and non-functional objectives ◦ Embedded/handheld software testing must include “the system” (hardware, software, operations, users, etc.)  Attacks go after common modes of failure and bugs to demonstrate that “does not meet” exists  We go after our enemy with many approaches Jon Hagar Copy right 2013 ◦ ◦ ◦ ◦ ◦ Tools Levels Attacks Techniques Etc. 38 Software Test Attacks To Break Mobile and Embedded Devices 19
  • 8/20/2013 An Attack Is…  Based on a common mode of failure seen over and over ◦ Maybe seen as a negative, when it really is a positive ◦ Goes after the “bugs” that may be in the software ◦ Based on or using classic test techniques and test concepts  Lee Copeland’s book on test design  Many other good books  A Pattern (more than a process) which must be modified for the context at hand to do the testing  Testers learn these in a domain after years and form a mental model (most good testers attack) Jon Hagar Copy right 2013 39 Software Test Attacks To Break Mobile and Embedded Devices Kinds of Attacks  Whittaker offers a good starting point for software attacks in general that can be applied to embedded: ◦ User Interface Attacks ◦ Data and Computation ◦ File System Interface ◦ Software/OS Interface  Whittaker’s “How to Break Software” lists 23 attacks  “Software Test Attacks to Break Mobile and Embedded Devices” (my book) adds 32 attacks and 8 sub attacks Jon Hagar Copy right 2013 40 Software Test Attacks To Break Mobile and Embedded Devices 20
  • 8/20/2013 Embedded Attack Classification          Developer Attacks (unit/code testing) Control System Attacks Hardware-Software Attacks Mobile and Embedded Software Domain Attacks Time Attacks (Performance) Human User Interface Attacks Smart and/or Mobile Phone Functional App Attacks Mobile/Embedded Security Attacks Generic Attacks ◦ Functional, mind mapping, and combinatorial tests Jon Hagar Copy right 2013 41 Software Test Attacks To Break Mobile and Embedded Devices The Software You Test  Do you know how it fails?  Do you test for success or failure or both?  Will this workshop give you all the answers and all possible attacks? ◦ No, but you can start asking questions and thinking Jon Hagar Copy right 2013 42 Software Test Attacks To Break Mobile and Embedded Devices 21
  • 8/20/2013 Introducing the Robots Requirements – in the class hand out for each robot grouping Rules (this exercise takes some thinking and reading) ◦ NO destructive testing (Please BE CAREFUL with the robots) ◦ There are bugs to be found (record and report them) ◦ Each group defines an attack and gets “time” on the devices (but time in our environment is limited—just as it is in the real world)  Environment ◦ This room, but we have some “test tools” ◦ This is software testing, but within the hardware it is embedded in  This will be a simple testing process, but use the attack concepts and report experiences back in the debrief 1. Define risks based on hardware, software, requirements, and bugs (risk list) 2. Conduct each attack session using a charter 3. Define a test attack using provided attack pattern (handout) 4. Will do a debrief after each test session 5. Group will rotate different robot configurations   Jon Hagar Copy right 2013 43 Software Test Attacks To Break Mobile and Embedded Devices Additional Considerations       We will try to run at least 2 different attacks Use the concepts we used on games Ask questions (of each other & me) No destructive testing and I am the “tester” (you must tell me what to do) Use the tools at hand But first we need to think about embedded users – next exercise Jon Hagar Copy right 2013 44 Software Test Attacks To Break Mobile and Embedded Devices 22
  • 8/20/2013 More Test Time        We are going to do another attack on a different embedded device Each group will be tasked with one of the two attacks Each group should complete a charter and see the “test master” for access to the device You’ll have a robot with software loaded Practice identifying: Risks, Users, Exploration, and Attacks Follow the “suggested patterns” of the attacks We’ll go until no time left Jon Hagar Copy right 2013 45 Software Test Attacks To Break Mobile and Embedded Devices Exercise: Understanding Users for Embedded Attacks  Let’s list some users of the games and robots because . . . ◦ Users play into risks, attacks, bugs, what to look for ◦ You should be able to do the same for the robots (or any software you test) Jon Hagar Copy right 2013 46 Software Test Attacks To Break Mobile and Embedded Devices 23
  • 8/20/2013 Exercise Answer: List Users Jon Hagar Copy right 2013 47 Software Test Attacks To Break Mobile and Embedded Devices Now you have a few basics: • • • • Jon Hagar Copy right 2013 Risk thinking Exploration Software’s users Attack patterns are provided next 48 24
  • 8/20/2013 Attack Group 15  Scenarios and actions over time Jon Hagar Copy right 2013 49 Software Test Attacks To Break Mobile and Embedded Devices Attack Stories, Tours, and Scenarios  Call them what you will  There are subtle differences depending on whose material you have read They are how the system gets used end-to-end They combine use, users, information, techniques, tools, and (maybe) attacks   Jon Hagar Copy right 2013 50 Software Test Attacks To Break Mobile and Embedded Devices 25
  • 8/20/2013 Apply This Attack When…   Time interacts with the software, events, inputs, and outputs Checklist of things to look for and consider (possible bugs) ◦ Order problems ◦ Too Long ◦ Too Fast ◦ Not at Right Time mark or point ◦ Late ◦ Late or early ◦ Early ◦ Deadlocked caused by a race condition(hard to find) ◦ Extra input or output events ◦ Missing events ◦ Wrong input/output within events Jon Hagar Copy right 2013 51 Software Test Attacks To Break Mobile and Embedded Devices Attack Factors  What - Look for things not in the right order  Who – Test team  Where – Lab and/or field testing where hardware and software interact ◦ Tools may be important here Jon Hagar Copy right 2013 52 Software Test Attacks To Break Mobile and Embedded Devices 26
  • 8/20/2013 How (a generic pattern)?  Understand what the system does or is supposed to do ◦ a sequence of events or functions ◦ Look in: concepts of operations, user guides, use cases, models, & any other information that will detail functions and usage over time ◦ From these, organize a sequence or set of sequences  First attack case: Focus on a typical situation based on requirements and/or use cases  Second attack case: Consider the off–normal, non–failure modes ◦ Look for the failure modes and effects—does the software recover well? ◦ Review and understand system errors and failure history from the field  Build up histories of attacks based on outputs and log files ◦ Warning: log files can contain large amounts of detailed data and this can also adversely affect the performance (especially timing) of the software  Conduct risk analysis as the effort progresses  Final Attack cases: Build Extreme cases such as “Soap Opera” Tests  Warning: Watch becoming “script” bound Jon Hagar Copy right 2013 53 Software Test Attacks To Break Mobile and Embedded Devices Exercise – Tell Me Stories   Work in groups Key points ◦ Define a “story/scenario” with a name and outline ◦ Use the check list on chart 51 (note what is used) ◦ Follow pattern of chart 52 (note steps on charter)  Can you build a Tour (combination of story patterns)  Products ◦ ◦ ◦ ◦    Risk list User list Test charter Note bugs (if any) Complete the feedback retrospective Do several attacks (note what each one is) Expand to “extreme cases” Jon Hagar Copy right 2013 54 Software Test Attacks To Break Mobile and Embedded Devices 27
  • 8/20/2013 Attack 7  Digitals v Analog Integration Jon Hagar Copy right 2013 55 Software Test Attacks To Break Mobile and Embedded Devices Attack Hw-Sw Interface Group Here we are attacking the hardware-tosoftware interface Many bugs - Developers hide in the interfaces often miss these Jon Hagar Copy right 2013 56 Software Test Attacks To Break Mobile and Embedded Devices 28
  • 8/20/2013 Attack: Analog-Digital Hw-Sw Interfaces  When - The software is “controlling” the unique hardware  What – Look at the interface, hardware (as a user), and what the software is controlling  Who – Test team (independent)  Where – Lab where the hardware and software are both present  Bugs to look for (next page) Jon Hagar Copy right 2013 57 Software Test Attacks To Break Mobile and Embedded Devices Taxonomy: A2D and D2A Bug Possibilities Type A2D A2D A2D D2A D2A D2A Situation Impact A2D representation information is lost Software because measurement is not precise computation is based on incorrect data A2D information is contaminated with Software noise computation use noise when it should not A2D information is calculated Computation has correctly unknown error D2A conversion losses “least significant bits” (LSB) in conversions, but bits are, in fact, important because computer word sizes are too small D2A information does not account for noise of the real world D2A information is calculated correctly because of internal factors Output to analog device is wrong Software computation does not include a factor for noise Computation has unknown error Notes Number of bits used to store the analog converted data is not large enough or sampling rate to get bits is not correct. The noise term may not be known, accounted for, or misrepresented. Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point Number of bits stored from the digital world to the analog world do not have enough precision, so analog data is incorrect. The analog values are not correct given the noise of the real world (output data may be lost in the noise). Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point Jon Hagar Copy right 2013 58 Software Test Attacks To Break Mobile and Embedded Devices 29
  • 8/20/2013 How?              Up front data gathering and analysis are important beginnings – what do we know (or can ask about) Identify input devices Identify output devices Define the input disturbances (unexpected system inputs) Define possible output disturbances (unexpected system outputs) Determine what is or is not possible in the test environment Conduct a risk analysis (see likely bugs table) Identify the users of the device and software - Testers should be aware that embedded systems have resource constraints in memory, CPU usage, and time Use the above information to define an exploratory chart attack Go run that attack Learn Design Repeat (until time runs out) Think! Jon Hagar Copy right 2013 59 Software Test Attacks To Break Mobile and Embedded Devices Questions to Ask with this Attack  If the hardware is a prototype (not like what will be in the field), will that impact testing or test results?  If a simulation is used, what bugs might be missed because actual hardware or software is not used?  If the test inputs and environment are not representative of the real world both in terms of expected and unexpected values, what risks will be acceptable?  If the hardware is not understood, will testing be weak?  If the major sources of “noise” is not defined, will the system be susceptible to impacts from unexpected inputs or outputs?  All of these questions will involve test tradeoffs, acceptable risk, and compromise ◦ 40% of this kind of attack should be “normal” situations ◦ Start with normal and move to off-normal and stress cases Jon Hagar Copy right 2013 60 Software Test Attacks To Break Mobile and Embedded Devices 30
  • 8/20/2013 Exercise – A2D/D2A   Work in groups Key points ◦ Think how to look for bugs of chart 58 ◦ Use the questions on chart 60 (note what is used) ◦ Follow pattern of chart 59 (note steps on charter)  Can you expand with stories/tours?  Products ◦ ◦ ◦ ◦    Risk list User list Test charter Note bugs (if any) Complete the feedback retrospective Do several attacks (note what each one is) Expand to “extreme cases” Jon Hagar Copy right 2013 61 Software Test Attacks To Break Mobile and Embedded Devices Feedback – Retrospective Session  What did you accomplish? ◦ Bugs? ◦ Tests?  What things did you think of? ◦ Wish I had a _____???? I need more time??? What favors or opposes an attack?  What would you do differently next cycle?  Jon Hagar Copy right 2013 62 Software Test Attacks To Break Mobile and Embedded Devices 31
  • 8/20/2013 Final Thoughts . . . Jon Hagar Copy right 2013 63 Wrap Up  This tutorial covers some basic introduction (key attacks) and sampling ◦ There are many more Understanding your local context and error patterns is important (one size does NOT fit all)  Attacks are patterns…you still must THINK!  These attacks target Embedded and Mobile  Jon Hagar Copy right 2013 64 Software Test Attacks To Break Mobile and Embedded Devices 32
  • 8/20/2013 More Attacks (from my book and others)  Attack 1: Static Code Analysis  Attack 18: Bugs in Timing Interrupts and Priority Inversion  Attack 2: Finding White–Box Data Computation Bugs  Attack 19: Finding Time Related Bugs  Attack 3: White–Box Structural Logic Flow Coverage  Attack 20: Time Related Scenarios, Stories and Tours  Attack 4: Finding Hardware–System Unhandled Uses in Software  Attack 21: Performance Testing Introduction  Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs Attack 22: Finding Supporting (User) Documentation Problems   Sub–Attack 22.1: Confirming Install–ability  Attack 6: Long Duration Control Attack Runs  Attack 23: Finding Missing or Wrong Alarms  Attack 7: Breaking Software Logic and/or Control Laws  Attack 24: Finding Bugs in Help Files  Attack 8: Forcing the Unusual Bug Cases  Attack 25: Finding Bugs in Apps  Attack 9 Breaking Software with Hardware and System Operations  Attack 26: Testing Mobile and Embedded Games  Attack 27: Attacking App–Cloud Dependencies 9.1 Sub–Attack: Breaking Battery Power  Attack 28 Penetration Attack Test  Attack 10: Finding Bugs in Hardware–Software Communications  Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack Attack 11: Breaking Software Error Recovery   Attack 28.2 Sub–Attack Fuzz Test  Attack 29: Information Theft—Stealing Device Data  Attack 12: Interface and Integration Testing  Attack 29.1 Sub Attack –Identity Social Engineering  12.1 Sub–Attack: Configuration Integration Evaluation  Attack 30: Spoofing Attacks  Attack 13: Finding Problems in Software–System Fault Tolerance  Attack 30.1 Location and/or User Profile Spoof Sub–Attack  Attack 14: Breaking Digital Software Communications  Attack 30.2 GPS Spoof Sub–Attack  Attack 15: Finding Bugs in the Data  Attack 31: Attacking Viruses on the Run in Factories or PLCs  Attack 16: Bugs in System–Software Computation  Attack 32: Using Combinatorial Tests  Attack 17: Using Simulation and Stimulation to Drive Software Attacks Attack 33: Attacking Functional Bugs   Jon Hagar Copy right 2013 65 Software Test Attacks To Break Mobile and Embedded Devices Summary: Thank You (ideas used from) James Whittaker (attacks) Elisabeth Hendrickson (simulations)  Lee Copeland (techniques)  Brian Merrick (testing)  James Bach (exploratory & tours)  Cem Kaner (test thinking)      Many teachers Generations past and future Books, references, etc. Jon Hagar Copy right 2013 66 Software Test Attacks To Break Mobile and Embedded Devices 33
  • 8/20/2013 Book List (my favorites)  “Software Test Attacks to Break Mobile and Embedded Devices”  “How to Break Software” James Whittaker, 2003 ◦ And his other “How To Break…” books      “Testing Embedded Software” Broeckman and Notenboom, 2003 “A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995  Honorable mentions: ◦ “Embedded System and Software Validation” Roychoudhury, 2009 ◦ “Systems Testing with an Attitude” Petschenik 2005 ◦ “Software System Testing and Quality Assurance” Beizer, 1987 ◦ “Testing Computer Software” Kaner et. al., 1988 ◦ “Systematic Software Testing” Craig & Jaskiel, 2001 ◦ “Managing the Testing Process” Black, 2002 – Jon Duncan Hagar, due out late 2013 (http://www.crcpress.com/product/isbn/9781466575301) Jon Hagar Copy right 2013 67 Software Test Attacks To Break Mobile and Embedded Devices More Resources • www.stickyminds.com – Collection of test info My Web site: www.breakingembeddedsoftware.com • Association of Software Testing • – BBST Classes http://www.testingeducation.org/BBST/ • Your favorite search engine Jon Hagar Copy right 2013 68 Software Test Attacks To Break Mobile and Embedded Devices 34
  • 8/20/2013 Definitions (for this class)             Taxonomy – the practice and science of classification. Test – the act of conducting experiments on something to determine the quality and provide information Test case – one set of inputs, environmental set up, and results (expected and unexpected) Attack – to set up, forcefully and attempt to “damage” the system or software, using tools, methods, and techniques Bug (error) – results that depart from the expected (from requirements, design, standards, user, etc.) Lifecycle – from beginning-to-end, the steps, stages, and activities to create (birth-to-death) Procedure – a particular way of accomplishing tests, usually written (one or more test cases) Tour – a journey to find information (tests) with a focus/direction (story) Scenario – a sequence of events with a test plot or story Script – see procedure, but normally uses automation Users – someone/something that interacts with the system/software (can be human or machine, or?) Quality – value to someone and that they will pay for Jon Hagar Copy right 2013 69 Software Test Attacks To Break Mobile and Embedded Devices Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: – Risk (s): Success Criteria: 1. – Attack 2. – Other (requirements, …..) • 3. • Support items needed: • Role (User you play during the test): Actions: – 1. – 2. – 3. • Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Jon Hagar Copy right 2013 70 Software Test Attacks To Break Mobile and Embedded Devices 35
  • 8/20/2013 Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: – Risk (s): Success Criteria: 1. – Attack 2. – Other (requirements, …..) • 3. • Support items needed: • Role (User you play during the test): Actions: – 1. – 2. – 3. • Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Jon Hagar Copy right 2013 71 Software Test Attacks To Break Mobile and Embedded Devices Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: – Risk (s): Success Criteria: 1. – Attack 2. – Other (requirements, …..) • 3. • Support items needed: • Role (User you play during the test): Actions: – 1. – 2. – 3. • Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Jon Hagar Copy right 2013 72 Software Test Attacks To Break Mobile and Embedded Devices 36
  • 8/20/2013 Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: – Risk (s): Success Criteria: 1. – Attack 2. – Other (requirements, …..) • 3. • Support items needed: • Role (User you play during the test): Actions: – 1. – 2. – 3. • Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Jon Hagar Copy right 2013 73 Software Test Attacks To Break Mobile and Embedded Devices Exploratory Test Card (Charter) Name of Test: Who is testing (test team) What to Test: – Risk (s): Success Criteria: 1. – Attack 2. – Other (requirements, …..) • 3. • Support items needed: • Role (User you play during the test): Actions: – 1. – 2. – 3. • Others Steps Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….) Jon Hagar Copy right 2013 74 Software Test Attacks To Break Mobile and Embedded Devices 37