Presentation
Upcoming SlideShare
Loading in...5
×
 

Presentation

on

  • 444 views

 

Statistics

Views

Total Views
444
Views on SlideShare
444
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Presentation Presentation Presentation Transcript

  • PROCESS … The GLUE that holds Software Projects Together A Focus on Software Testing Dennis A. Morrison Senior Principal Engineer DD(X) System Architecture [email_address]
  • Presentation Outline
    • Software Process Overview
    • Software Testing Overview
    • Source Code Control
    • Trouble Reporting and Bug tracking
    • Summary
  • Software Phases (Waterfall Model) Requirements Design Conceive Code Test Build Deploy Support Deliver
  • Software Phases (Process Glue) Requirements Design Code Test Deploy Support Requirements Review – Are the Requirements testable Artifacts – Review documentation; Requirements placed under documentation control Design Review – Does the design support requirements? Artifacts – Review documentation Code Review – Does the code support requirements and implement the design?? Artifacts – Review documentation; Code under source code control Test Review – Test plans Test Results – Did the tests pass? Artifacts – Review documentation; Test plans; System Trouble Reports (STR) Test Review – Customer Acceptance Test Test Results – Did the tests pass? Artifacts – Review documentation; STRs Customer Survey – Is the system meeting customer expectations Artifacts - Software Updates; STRs
  • Software Phases and Loops Requirements Design Code Test Deploy Support Phases where most rework occurs New feature requests or feature refinement Design Errors Software Bugs Software Bugs New Feature Request Requirement Clarification or changes
  • Software Testing An Art or a Science ?
    • Testing is a phase of software development that can become, or seem, endless
      • Classically, the most expensive part of the project
      • Schedule overruns are often attributed to the testing cycle
      • Testers are usually squeezed into smaller and smaller test cycles
    • How much testing is enough?
    • How do you know when you are finished testing?
    • How good is the product when you are finished?
  • Testing Has Many Stages Test Unit Test White Box Testing Integration Test Grey Box Testing System Test Black Box Testing Acceptance Test Subset of Black Box Testing
    • Phase = Unit Test
    • Testing is focused on internal operations of the software component
      • Inputs/outputs
      • Logical operations
      • Limit testing
      • Limited functional testing with stubs
    • Testing methodologies
      • Formal testing with test plan and review
      • Ad-hoc random
      • Typically, tests are manually executed
    Unit Test = White Box Testing
    • Phase = Integration Test
    • Testing is focused on functional aspects of an integrated set of software components
      • Inputs/outputs
      • Logical operations from outside of the software components
      • Limit testing (induced error paths from external stimuli)
      • Stubs are replaced by real software as they become available
    • Testing methodologies
      • Formal testing with test plan and review
      • Semi-automated and manually executed
      • Automation yields regression test capability
    Integration Test = Grey Box Testing
    • Phase = System Test
    • Testing is focused on system level requirement verification
      • User Interface (Human to Computer Interface)
      • Requirement verification
      • Limit and normal operational testing
      • Stress and longevity testing
    • Tests are executed on target hardware and software
    • Testing methodologies
      • Formal testing with test plan and review
      • Typically, tests are semi automated and manually executed
      • Automation yields regression test capability
    System Test = Black Box Testing
    • Phase = Acceptance Test
    • Testing is focused on externally visible features
      • User Interface (Human to Computer Interface)
      • Operational focus
      • Interfaces with external systems in customer domain
        • Other collaborative systems
        • Customer network, etc.
    • Tests are executed on target hardware and software usually at the customer’s site
    • Testing methodologies
      • Formal testing with test plan and review
      • Typically, tests are manually executed
    Acceptance Test = Subset of Black Box Testing
  • Some Software Testing Models
    • Developer level testing
      • Applicable to small teams
    • Community testing
      • Beta versions of a product are released to a volunteer community for functionality and user level testing
        • Used by gaming community prior to product releases
        • Microsoft uses this approach as one of their test phases
    • Collaborative testing
      • Developers and customers test together
    • Project level testing
      • Used by most large projects
  • When Should Software Testing Begin?
    • The software test cycle should begin during the project requirements phase
    • Requirements review should result in the first level of test specifications
    • Similar to the “Agile” software development process, test cases should be written and reviewed before the software is developed
  • When is Testing Finished?
    • When the schedule says you’re finished?
    • When you run out of money?
    • When the boss says to release it?
    • When the tester’s say you’re done?
    • When the customer says you’re done?
    • When the law suits are filed?
  • Requirements Verification
    • Raytheon uses requirement verification and statistical mechanisms to determine when testing is complete
    • Requirements are decomposed and allocated to software components
    • Requirements are “sold off” with during formal testing events with stakeholders (customers) present
  • Requirement Decomposition Req 1 Operational Requirement Parent requirement decomposes into three child requirements The system shall provide three factor user authentication Something you have Token Reader Something you know Pin Reader Something you are Biometric Reader Req 1.3 Req 1.2 Req 1.1 Req 1.3.2 Req 1.3.1 Req 1.2.1 Req 1.1.2 Req 1.1.1
  • Requirements Role in Testing
    • Software is tested against requirements
      • Before a parent requirement can be “sold off”, all children requirements must be fulfilled
      • Before a child requirement can be “sold off”, all of its siblings must themselves be “sold off”
      • The failure of any requirements will prevent the parent from being fulfilled
  • Requirement Decomposition Req 1 Operational Requirement Parent requirement decomposes into three child requirements The system shall provide three factor user authentication Req 1 Something you have Token Reader Something you know Pin Reader Something you are Biometric Reader Req 1.3 Req 1.2 Req 1.1 Req 1.3.2 Req 1.3.1 Req 1.2.1 Req 1.1.2 Req 1.1.1 Req 1
  • Requirements Traceability
    • Software test cases are built from system use cases
      • Scenarios (rainy day and sunny day) are used to test various threads through the software
      • Every scenario is traced to parent requirements
        • A scenario may have more than one parent requirement
      • Test cases must demonstrate interface level testing
        • All system and software level interfaces are documented and tracked
        • These are known as IRLLs, IRLEs, and IRLMs
          • IRLL = Interface Requirement Logical Level
          • IRLE = Interface Requirements Electrical Level
          • IRLM = Interface Requirements Mechanical Level
    • Test cases and requirements are stored in DOORS
  • Software Source Code Control
    • The ability to save and retrieve versions of software is a key component to success in medium to large software projects
      • Software can be “versioned” for new features or software load lines
      • Changes made to software modules due to bug fixes can be isolated
      • Enables developers to work on next release while maintaining current versions
      • Software builds can include or exclude changes
  • Using Source Control Systems Source Code Version 1 Source Code Control System Source Code Version 1 Put (ModA V1) get(ModA V1) Source Code Version 2 put(ModA V2) Mod A-V1.1 Mod X-V7 Mod Z-V2 SW Build Specification Software Build Source Code Version 1.1 Source Code Version 1 get(ModA V1) put(ModA V1.1) New Feature Bug Fix
  • Trouble Reporting and Bug Tracking
    • Software bugs are documented using formal mechanisms
      • CMTS – Change Management Tracking System (MRs)
      • WSTR – Web System Trouble Report (STRs)
    • Bugs are considered addressed when software has been modified, tested, and submitted to the load-line
    • Bugs are considered fixed when they have been verified
    • Bug fixes are pulled into the software load via the source code control system
    • Bug reports and time based metrics guide management in software release decisions
  •  
  • Test Methods
    • Manual
      • Usually applied during unit test
    • Semi Automated
      • Applied during integration test
    • Fully Automated
      • System test
  • Test Automation
    • Test automation can provide a project with huge dividends
    • Test scripts
      • Drive test automation engines
      • Can be used in unit testing
      • Can be reused by multiple test organizations
    • Regression tests can be constructed from reusable test scripts
  • Metrics Drive Behavior
    • An organization or individual’s behavior is directly related to the metrics they are measured by
      • Defects found and fixed
      • On time delivery
      • Productivity
        • SLOC counts
        • Defect rate at delivery
      • Complexity indices
      • Customer satisfaction
  • Best Practices?
    • No real silver bullets
    • Process is the glue that holds a project together
    • The right solution depends on the project but is often a combination of methods/processes and tools
    • Just remember that customer satisfaction trumps all!!
  • BACKUP SLIDES
  • In the world of software development, process is the “glue” that holds the project and team together. Too little, the pieces fall apart; too much and it can become a sticky quagmire resulting wasted effort, time, and money. The challenge facing many projects is finding just the right balance point on the process continuum. As a project progresses through its life cycle, the balance point will shift and the project will need to continuously review and adjust to maintain optimal performance.   A typical software project life cycle is comprised of six distinct phases: Requirements, Design, Code, Test, Deployment, and Support. Most projects place a significant amount of emphasis on the front-end phases of the life cycle e.g. requirements, and design. However, industry data clearly demonstrates that the largest percentage of a projects budget is spent during the final phases, namely test, deployment, and support. These phases encapsulate much of the “rework” experienced by the project.   This talk will describe the process philosophies and tools used by Raytheon to tackle large-scale multisided software development projects.   ABSTRACT
  • Tools
    • The right tool can result in huge pay backs
    • Source Code Control Tools
    • Trouble Report tracking
    • Stress testing
    • Performance testing
  • Testing Methodologies
    • Ad-hoc
      • Non structured testing with or without test artifacts
      • No test plans
      • Non repeatable tests
    • Formal testing
      • Test plans
      • Test Artifacts
      • Defect tracking
  • Software Phases (Spiral Model) Spiral Development: Experience, Principles, and Refinements Spiral Development Workshop February 9, 2000 Barry Boehm, edited by Wilfred J. Hansen
  • The NATO Sea Sparrow Surface Missile System (NSSMS) is a quick reaction, point defense system designed to protect the ship it is on. It is very, very effective against enemy aircraft, low flying sea skimming missile and even bomb laden inflatable boats. It was first developed by Raytheon in the '70's and installed on over 150 ships world-wide. In the mid '90's the Navy contracted Raytheon to re-architect the system so that it would be capable of firing the Navy's newest missile the Evolved Sea Sparrow Missile (ESSM). The Navy's brand new nuclear powered aircraft carrier , the USS Ronald Reagan, has 6 Seasparrow radars & 3 Seasparrow missile launchers. The movie shows the Navy doing Qualification Testing on these systems.
  • Software Phases (Process Glue) Requirements Design Code Test Deploy Support Requirements Review – Do the requirements support the customer’s intent? Are they testable? Are they complete (sunny day/ rainy day) Artifacts – Review documentation; Requirements placed under documentation control Design Review – Does the design support requirements? Artifacts – Review documentation Code Review – Does the code support requirements and implement the design?? Artifacts – Review documentation; Code under source code control Test Review – Test plans – Do the tests adequately exercise the requirements? Test Results – Did the tests pass? Artifacts – Review documentation; Test plans; System Trouble Reports (STR) or Modification Requests (MR) Test Review – Customer Acceptance Test Test Results – Did the tests pass? Artifacts – Review documentation; System Trouble Reports (STR) or Modification Requests (MR) Customer Survey – Is the system meeting customer expectations Artifacts - Software Updates; STRs